Ai-powered tracker based on imaging data for end-organ ultrasound stimulation therapies

Abstract

A method of tracking an organ, including: receiving an image from a time series of images of the organ collected by an imaging device; selecting a region of interest (ROI) of the image and identifying features within the ROI; tracking the identified features from the image to a subsequent image collected later in the time series; determining a confidence level of the tracking and assessing whether the confidence score is acceptable, in which, if the confidence level is not acceptable, a trained network is used to predict a location of the ROI in a set of subsequent images collected over a time range, and resuming tracking and determining the confidence level of the tracking after the time range; and outputting a trajectory of the organ over the time series of images.

Description

UMN 2025-187-02Quarles 920171.00690 AI-POWERED TRACKER BASED ON IMAGING DATA FOR END-ORGAN ULTRASOUND STIMULATION THERAPIES CROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The present application is based on and claims priority from U.S. Patent Application Ser. No. 63 / 746,463, filed on January 17, 2025, the entire disclosure of which is incorporated herein by reference.BACKGROUND

[0002] Recent research reveals the potential of ultrasound (US) stimulation of spleen and liver for treating inflammatory diseases and diabetes, respectively. Noninvasive tools for modulating neural and cellular signaling pathways in peripheral organs will advance the understanding of the role of these pathways in maintaining health and influencing diseases. While promising results have been observed in this technique, challenges remain, such as failures in nerve activation and off-target stimulation, which can reduce treatment efficacy and increase the risk of affecting surrounding tissues. In addition, the motion of abdominal organs during respiration can hinder accurate targeting.SUMMARY OF THE INVENTION

[0003] Accordingly, new systems, methods, and apparatus for tracking motion of an organ are desirable.

[0004] In one embodiment, the invention provides methods and systems for tracking motion of an organ, including receiving an image from a time series of images of the organ, collected by an imaging device; selecting a region of interest (ROI) of the image and identifying features within the ROI; tracking the identified features from the image to a subsequent image collected later in the time series; determining a confidence level of the tracking and assessing whether the confidence scores is acceptable, in which, if the confidence level is not acceptable, a trained network is used to predict the ROI’s location in a set of subsequent images collected over a time range, and resuming tracking and determining the confidence level of the tracking after the time range; and outputting a trajectory of the organ over the time series of images.

[0005] In some embodiments, the ROI may contain texture information from which the features can be identified based on a gradient distribution of a structure tensor of the ROI.

[0006] In some embodiments, the trained network used to predict the location of the ROI in a set of images over a time range includes a long short-term memory (LSTM)UMN 2025-187-02Quarles 920171.00690 algorithm, configured to have an architecture comprising layers and hidden units.

[0007] In some embodiments, the time range over which the location of the ROI is predicted is equal to or greater than a duration of one breath cycle.In some embodiments, the systems and methods further include adjusting the location of the ROI to a location with a greater number of features for tracking.

[0008] In some embodiments, the systems and methods further include delivering a therapy to the organ with a therapy device comprising at least one of a motion sensor, vibration sensor, respirometer, heart rate sensor, the imaging device, or another imaging device.In some embodiments, the organ includes an abdominal organ, and the abdominal organ is selected from a group including liver, spleen, stomach, small intestine, large intestine, gallbladder, pancreas, adrenal gland, uterus, fallopian tubes, ovaries, kidneys, ureters, and bladder.

[0009] In some embodiments, multiple ROIs are tracked. In some embodiments, multiple organs are tracked.

[0010] In some embodiments, assessing whether the confidence level is acceptable include determining a confidence score and assessing whether the confidence score is greater than a threshold. In other embodiments, assessing whether the confidence level is acceptable includes receiving user input indicating whether the confidence level is acceptable.

[0011] In some embodiments, the imaging device includes at least one of an ultrasound device, a magnetic imaging device, or a fluoroscopy device.BRIEF DESCRIPTION OF THE DRAWINGS

[0012] Various objects, features, and advantages of the disclosed subject matter can be more fully appreciated with reference to the following detailed description of the disclosed subject matter when considered in connection with the following drawings, in which like reference numerals identify like elements.

[0013] FIG. 1 shows US data collection for the spleen. (Left) Transducer placement, and (Right) lateral view of spleen in US image with US transducer positioned at top of image with blue dot corresponding to raised circle location in (left).

[0014] FIG. 2 shows spleen manual annotation. All frames in each sequence were examined to identify visible and trackable landmarks on the hilum. The red boxes represent the ROIs around these landmarks in the first frame of each sequence. Three to four points were selected within these ROIs to extract their motion trajectories as the spleen moves.UMN 2025-187-02Quarles 920171.00690

[0015] FIG. 3 shows examples of liver annotation on US images. The red boxes show the ROIs on the first frame of each US sequence. Three to four points (pixels) inside the ROIs are selected as landmarks for tracking.

[0016] FIG. 4 shows that the proposed end-to-end tracker uses a combination of an enhanced KLT tracking and LSTM prediction framework.

[0017] FIG. 5 shows a MR-EKLT tracking process on a frame sequence of the spleen. First column shows two selected ROIs in red and blue color on the first frame of the same sequence. The second column demonstrates detected features by the algorithm as good features for tracking, and third column shows the features among selected ones that satisfied the confidence conditions after 10 cycles of breathing.

[0018] FIG. 6 shows a MR-EKLT tracking process on the liver. First column shows two selected ROIs in red and blue color. The second column demonstrates detected features as good features for tracking, and third column shows the features among selected ones which satisfied the confidence conditions after 5 minutes and 28 seconds of breathing.

[0019] FIG. 7 shows examples of two US sequences during deep breathing, (a) Five consecutive frames from a sequence where the red ROI, defined on the spleen’s hilum, is occluded by the lung in frames 3 and 4 but becomes visible again in frame 5. The blue ROI near the caudal point of the spleen remains visible throughout all shown frames, (b) Nine consecutive frames from another participant, where the red ROI is lost from frame 3 to frame 7 and reappears in frame 8. Predicted red ROIs are shown in frame 5 of (a) and frame 8 of (b) using the proposed LSTM to handle occlusion cases.

[0020] FIG. 8 shows architecture of the prediction model for features (ROI) prediction. The input layer consists of 5 sequential time steps. LSTM layer consists of 128 hidden units. A fully connected layer takes output LSTM’s output, and regression layer predicts the location of the selected features.

[0021] FIG. 9 shows normalized and detrended motion of the spleen in cm for red and blue ROIs shown in FIG. 7, using the MR-EKLT algorithm in one-step with an accuracy greater than 0.98 for detected features.

[0022] FIG. 10 shows sequence categories based on applied algorithm. 24 US sequences including 9 liver sequences and 15 spleen US sequences could be tracked by EKLT tracker. 25 sequences could not be tracked by EKLT tracker in one-step, and they needed new ROI definitions and as a result the EKLT-LSTM algorithm applied for tracking.

[0023] FIGS 11A-11E shows EKLT-LSTM tracking and prediction. FIG. 11A. TwoUMN 2025-187-02Quarles 920171.00690 ROIs (red and blue) close to hilum were defined in the first frame of the sequence. FIG. 11B. EKLT tracker followed reliable features during respiration and spleen motion based on confidence condition in the two ROIs. FIG. 11C. The frame in which the EKLT missed the features. FIG. 11D. Predicted locations for the best feature after 10 frames following the occurrence of the missing features. FIG. 11E. Defined ROIs with center located on the predicted locations.

[0024] FIG. 12 shows a flow chart of a method to track an organ.

[0025] FIG. 13 shows a schematic diagram of an exemplary computer system.DETAILED DESCRIPTION

[0026] Recent neuromodulation research has revealed the potential of ultrasound (US) stimulation for treating a range of health disorders through nerve stimulation. The stimulation of the liver using US has demonstrated its capability to modulate pathways responsible for regulating blood glucose levels, which is helpful to treat diabetes disorders and obesity. Additionally, US stimulation of the spleen can activate the cholinergic antiinflammatory pathway and parallel inflammatory pathways and reflexes within the spleen, which have been shown to reduce cytokine release and inflammation. This outcome could be beneficial in treating various inflammatory diseases, such as rheumatoid arthritis. While promising results have been observed using US stimulation, improvements in neural and cellular activation and reduction of off-target stimulation can lead to greater treatment efficacy.

[0027] The current targeting adjustment process is performed manually; it starts with an initial US scan to image the targeted organ. The US transducer is then positioned on the skin over the organ. Once the organ is identified, the transducer’s position is marked on the skin for use during the procedure. A key limitation of this stimulation technique is the lack of automated, intelligent tools for precise US targeting of the region of interest (ROI) containing necessary cellular pathways in real-time, capable of automatically monitoring any changes. Additionally, daily administration over weeks adds to the complexity, requiring repeated setup and increases the expenses. Other factors such as skeletal structure and organ location further complicate targeting. Notably, these organs can move due to breathing or body movement. In various stages of fitting the device on the body to make sure that the ultrasound can continue to target the organ, it is critical to be able to track the motion of the organ to sufficiently characterize the motion and in an automated way or semi-automated way forUMN 2025-187-02Quarles 920171.00690 efficiency. The systems and methods described herein explain the development of novel steps and algorithms including Al algorithms to enable automatic or semi-automatic tracking of organ motion, such as spleen and liver (but can be applicable to many other organs or objects within the body), to then provide its motion trajectory as well as different anatomical boundaries and features of the organ even when not moving. In a practical implementation, you can imagine that someone comes into a clinic for a fitting session where the clinician or technician places an ultrasound imaging probe on the body to detect the spleen (or other organ) and collects imaging data for a few minutes and under different breathing conditions (e.g., have the patient do deep breathing and soft breathing and fast breathing and stand up or lay down). That data can then be processed and inputted into an algorithm that will then provide a trained motion model that shows how that organ or ROI is shaped and its motion under different conditions. This data-driven model can then be inputted into a wearable device that uses it to track the motion of the spleen using multiple types of detection sensors (e.g., accelerometer, respiratory belt, heart rate sensor, ultrasound signals, optical signals, mechanical or magnetic sensors for stretch or motion) and algorithms to then apply an ultrasound beam to target the spleen over time. This is just one example of how it can be implemented. The invention is not only the algorithm and steps in obtaining the images and algorithm but also combining it with different conditions like have the patient do different breathing patterns and in different physical positions (laying down, standing up, on the side, etc.) to get various conditions since more data of different conditions provides more generalizability of the algorithm to know how the organ moves and its shape also changes. All this information then can be used for a wearable device to enhance its ability to track the motion of the organ for stimulation targeting with ultrasound (or other modes of stimulation like magnetic stim or laser stim or other focused stim approaches). One challenging in US imaging is noisy images, occlusions, rib shadows, and out-of-plane motion. In the current clinical setting for US tracking, tracking is paused to redefine the ROI, the methods and systems described herein disclose the development of a deep learning-based predictor integrated into the motion model to allow continuous tracking without interruptions. These methods can be applied in various image-guided treatments. Zebarjadi, M., et al, IEEE Transactions on Biomedical Engineering, vol. 72, no. 12, pp. 3484-3500, 2025 is herein incorporated by reference in its entirety.

[0028] Getting into more details of the approach, the novel invention involves incorporating an automatic framework comprising a highly accurate tracking method and anUMN 2025-187-02Quarles 920171.00690 efficient prediction approach in the future US stimulation devices. This invention has been approached from various angles: (1) The automatic approach allows for the initial definition of the ROI of the organ, which contains neural inputs into it or high blood vessel density (or whatever is the key feature relevant for the desired target region), in the first frame of the imaging sequence. Subsequently, the selected region is tracked and stimulated throughout the entire US imaging. This method provides more effective treatment in a shorter amount of time while minimizing exposure to surrounding tissues. (2) The tracker is an enhanced and modified version of the KLT tracker, which is well-suited for overcoming the challenges posed by traditional tracking methods in US domain, such as high levels of noise, and is robust to varying levels of tissue deformation. It has been enhanced in several ways, notably by providing sub-pixel accuracy (error lower than 1 mm in normal breathing). Performance of 2 mm or lower has been considered satisfactory in previous studies. Such sub-millimeter accuracy is typically required for high-precision applications such as radiation therapy and image-guided surgery. In contrast, our application focuses on tracking a target region rather than a precise anatomical point and therefore does not require this level of accuracy.Nevertheless, the achieved performance demonstrates that the proposed approach not only meets the requirements of our application but also has the potential to be extended to more demanding, high-accuracy clinical applications. (3) The combination of the tracking with prediction method provides the capability to compensate for the implementation delay of the device. (4) Employing this deep learning combination, the device can overcome common hindrances in US imaging, such as organ occlusion, out-of-plane challenge, and rib shadowing, and larger deformations that can lead to tracking failure. Overcoming these challenges has been crucial in utilizing US as a motion dataset. For this purpose, the deep learning predictor, which is optimized LSTM network, is able to forecast the ROI's next location over a configurable prediction horizon (e.g., the next 2-3 seconds), enabling continued tracking during temporary occlusions until the region of interest becomes visible again.. This way, the tracker can jump from noisy or occlusion situation to improved one. Previous studies have not addressed this type of dataset. (5) Another interesting outcome of using the predictor is that it can adjust the location of the ROI to a better location to contain useful information for tracking. This is beneficial for studies aiming to determine the motion model of the organ.

[0029] This observation opens up the possibility for the development of various novel methods. US imaging has the potential to offer superior temporal resolution in capturingUMN 2025-187-02Quarles 920171.00690 dynamic organ movements, coupled with cost-effectiveness, real-time capabilities, and safety has made it a popular modality in quantitative applications. However, there is limited number of studies dedicated to the using of US images for model development due to challenges explained above. Using this framework, a large US dataset of organs' motion can be achieved and can be used in future studies for localization and motion estimation. In the realm of medical imaging, such datasets are often lacking. When medical images are accessible, creating high-quality models is challenging due to the complexity and cost of precise annotation. The proposed model is an alternate approach to minimize or eliminate the annotation process. All the methods and enhancements not only apply to end-organ US stimulation therapies but also hold relevance for a wide range of applications, including organ-targeted therapies (radiation therapy, robotic-assisted surgeries, image-guided interventions), diagnostic imaging, and real-time calibrations.

[0030] This is a new framework of use and software code. To our knowledge the tracker has not been implemented before for the US application to determine therapeutic energy targeting location on spleen and liver. Additionally, to the best of our knowledge, the common solution for dealing with high noise and occlusion (e.g., rib blocking the view of the spleen or other organ blocking the view of the liver) in other applications is to stop the tracking and define a new region of interest (ROI) but this framework can overcome this problem by integrating the predictor to its enhanced tracker by skipping this situation without interrupting the tracking process.

[0031] Liver US stimulation has demonstrated its ability to treat diabetes, obesity, and metabolic disorders. US stimulation of the spleen can also help to treat a wide range of health disorders, spanning rheumatoid arthritis, inflammatory bowel disease, acute kidney injury, myocarditis, pulmonary hypertension, and pancreatitis and many other disorders. Inspiring from these findings, a novel stimulation device powered by a tracker, which is centered on the modulation of electrical signaling patterns within the peripheral nervous system, provides an efficient treatment procedure. Noninvasive devices for modulating cellular and neural signaling in peripheral organs will advance the study of their role in maintaining health and influencing diseases.

[0032] In addition, a broad spectrum of medical image applications such as radiation therapy, robotic-assisted surgeries, diagnostic imaging, and image-guided interventions is impacted by the challenge of organ motion, especially during respiration. This motion leads to image artifacts, decreased diagnostic precision, misalignment with healthy surroundingUMN 2025-187-02Quarles 920171.00690 tissues, and compromised therapeutic efficacy. This tracking technique is a valuable technique to overcome motion complications.

[0033] All in all, the methods and enhancements not only apply to end-organ US stimulation therapies but also hold relevance for a wide range of applications, including organ-targeted therapies and real-time calibrations for other imaging applications for diagnostics.

[0034] The primary purpose for developing these methods was to target spleen accurately to stimulate it by US. US directed at the spleen decreased inflammation in inflammatory arthritis. However, delivery of neuromodulatory US to abdominal organs is complicated due to motion-related artifacts such as high noise, occlusion, and organs moving off-screen due to their high amplitude of motion. This study proposes a low-cost, safe, and an efficient automatic solution to determine ROI on spleen (potentially other organs too) and to track it with high accuracy up to the final frame of the sequence, even in the presence of high noise and occlusion.

[0035] The project is presently conducting tests on a larger participant group to incorporate various breathing patterns and body positions, as these factors can influence the speed of organ motion and its deformation. The study, initially based on data from 100 participants, aims to expand its scope to encompass a more diverse range of demographic data. After training and testing this framework on this large dataset, the intended application could leverage these methods to be implemented in real-time on the US therapy device.

[0036] In accordance with some embodiments of the disclosed subject matter, mechanisms (which can include systems, methods, and apparatus) for tracking an organ are provided.

[0037] In some embodiments, a method of tracking motion of an organ includes receiving an image from a time series of images of the organ collected by an imaging device. An imaging device may be an ultrasound device, MRI, x-ray, fluoroscopy, or other such medical imaging devices. The time series of images may be collected at a specific sampling frequency. The sampling frequency may be selected to capture rapid movements.

[0038] In some embodiments, the method further includes selecting a region of interest of the image and identifying features within the ROI. An ROI can be selected manually (e.g., a user selects the region) or automatically (e.g., using an algorithm or a trained network). The ROI may range in size from 2 to 30 mm.

[0039] In some embodiments, the ROI may include texture information. TheUMN 2025-187-02Quarles 920171.00690 identified features within the ROI may be identified using the texture information. For instance, a gradient distribution of a structure tensor of the ROI may be determined to identify features.

[0040] The method may further include tracking the identified features from the image to a subsequent image collected later in the time series, which can leverage different types of tracking algorithms. In some embodiments, tracking the identified features may be completed using an enhanced Kanade-Lucas-Tomasi (EKLT) method.

[0041] The method may further include determining a confidence level of the tracking and assessing whether the confidence level is acceptable. An “acceptable” confidence level refers to a confidence level that is within a desired range (e.g., higher than a determined threshold). An “unacceptable” confidence level refers to a confidence level that is outside of a desired range (e.g., lower than a determined threshold). A confidence level may include determining a confidence score of each identified feature in an ROI between a first frame and a subsequent frame. If there is a strong match between the neighborhoods of a specific feature, the feature has a higher confidence score.

[0042] In some embodiments, the confidence level is assessed manually (e.g., a skilled user assesses whether the confidence level is high enough to continue tracking, in which case the confidence level is acceptable). If the confidence level is determined manually, a user can input a value indicating whether the confidence level is acceptable or unacceptable. In some embodiments, the confidence level is assessed automatically (e.g., automatically compared to a pre-defined threshold; if the confidence level is greater than the threshold, the confidence level is acceptable).

[0043] In some embodiments, a confidence level may be determined based on the number of features with a confidence score above a score threshold. Features that have a confidence score below the score threshold may be excluded from analysis. In this embodiment, the confidence level threshold may refer to a number of features with a high confidence score (e.g., the threshold is 10 features with a high confidence score; if there are only 6 features with a high confidence score, the confidence level is “unacceptable”).

[0044] In some embodiments, if the confidence level is not acceptable, a trained network is used to predict a location of the ROI in a set of subsequent images collected over a time range, and resuming tracking and determining the confidence level of the tracking after the time range.

[0045] A trained network may include, but is not limited to, one or more of a neuralUMN 2025-187-02Quarles 920171.00690 network, recurrent neural network, long short-term memory (LSTM) algorithm, gated recurrent units, transformers, artificial intelligence, as well as deep learning algorithms or linear and nonlinear regression models.

[0046] In some embodiments, the trained network may include an LSTM algorithm. The LSTM algorithm may be configured to include an architecture including layers and hidden units.

[0047] In some embodiments, the time range may be about equal to or greater than the duration of a breath cycle. For instance, the time range may be between 1 second, 2 seconds, 3 seconds, 4 seconds, 5 seconds, 6 seconds, 7 seconds, 8 seconds, 9 seconds, or 10 seconds.

[0048] In some embodiments, the method further includes outputting a trajectory of the organ over the time series of images.

[0049] In some embodiments, the methods may further include adjusting the location of the ROI to a location with a greater number of features for tracking.

[0050] In some embodiments, the method may further include delivering a therapy with the imaging device. The therapy may include, but is not limited to radiotherapy, US stimulation, electrical stimulation, magnetic stimulation, optical stimulation, or neuromodulation therapy. In some embodiments, therapy may be administered real-time by using a surrogate detection sensor. Prior / pre-operative US imaging data and alignment can be used, and simple sensors can then be leveraged to track organs (e.g., acceleromoeters, vibration sensors, respirometers, heart rate trackers), which in some embodiments may allow therapy to be administered in real-time.

[0051] In some embodiments, the organ is an abdominal organ. The organ may include, but is not limited to, liver, spleen, stomach, small intestine, large intestine, gallbladder, pancreas, adrenal gland, uterus, fallopian tubes, ovaries, kidneys, ureters, and bladder. Multiple organs may be tracked.

[0052] Examples

[0053] Recent research reveals the potential of ultrasound (US) stimulation of spleen and liver for treating inflammatory diseases and diabetes, respectively. Noninvasive tools for modulating neural and cellular signaling pathways in peripheral organs will advance the understanding of the role of these pathways in maintaining health and influencing diseases. While promising results have been observed in this technique, challenges remain, such as failures in nerve activation and off-target stimulation, which can reduce treatment efficacyUMN 2025-187-02Quarles 920171.00690 and increase the risk of affecting surrounding tissues. In addition, the motion of abdominal organs during respiration can hinder accurate targeting. We have introduced a novel tracking framework called EKLT-LSTM to track liver and spleen using US imaging, which is a combination of an enhanced Kanade-Lucas-Tomasi (EKLT) tracker and a long short-term memory (LSTM) predictor. This integration leverages the strengths of both methods to improve the overall accuracy and robustness of the tracking system, where the EKLT tracker provided precise annotations that enhanced the training of the LSTM model. Simultaneously, the LSTM could handle situations where the EKLT faces US complications, such as occlusions and high level of noise, by utilizing the annotated data to make predictions. The LSTM block also has the capability of region of interest (ROI) adjustment. A total of 40 recordings collected from 10 participants were tested to evaluate the performance of the EKLT-LSTM on spleen motion tracking. Each participant underwent four separate recordings, each in a distinct breathing pattern to capture a diverse range of motion scenarios and different deformation rates, measured as rigid organ error (ROE). The method was also assessed on a liver motion dataset collected in previous studies from 9 subjects. The results demonstrated that the spleen tracking model was most accurate during slow, shallow breathing, with an averaged error of 0.4 ± 0.4mm, and that the maximum averaged error was about 1.37 ± 0.9mm during fast, deep breathing. Liver tracking results also showed high accuracy with averaged error of 0.3 ± 0.2mm. The proposed EKLT-LSTM framework yields advantages over previous tracking models for novel therapeutic applications in the emerging fields of end-organ modulation and bioelectronic medicine.

[0054] A broad spectrum of medical image applications, such as radiation therapy, robotic-assisted surgeries, diagnostic imaging, and image-guided interventions is impacted by the challenge of respiratory-induced motion (RIM). RIM can lead to image artifacts, decreased diagnostic precision, misalignment with healthy surrounding tissues, and compromised therapeutic efficacy. Motion tracking is a valuable technique to overcome RIM complications. Many tracking and prediction models ranging from linear filters to advanced deep learning algorithms have been applied to estimate the position of the target. Devices for RIM tracking have been suggested by key technology manufacturers, some of which are already in use. Extensive literature reviews have assessed the advantages and disadvantages of these approaches. These studies demonstrate the versatility of motion tracking technologies across various medical disciplines for improving precision, reducing invasiveness, and enhancing patient outcomes. In addition to these established treatments,UMN 2025-187-02Quarles 920171.00690 emerging therapeutic modalities are gaining prominence, necessitating a fresh perspective of these approaches. Particularly in the fields of peripheral nerve and end-organ modulation, it has been demonstrated that non-invasive ultrasound (US) stimulation targeting different splenic nerve and spleen regions can activate the cholinergic anti-inflammatory pathway or reflex anti-inflammatory pathways up to the brain and back down to the spleen, which can reduce cytokine release and inflammation. This US-based approach could help to treat a wide range of inflammatory diseases, such as rheumatoid arthritis. Recently, liver US stimulation has demonstrated its ability to modulate pathways responsible for regulating blood glucose levels, which has potential to treat diabetes and obesity disorders. Like other medical targeting applications mentioned earlier, nerve or end-organ stimulation presents several challenges in that those previous studies, for example, the response occurred only when specific sub-organ regions containing glucose-sensing neurons in liver were targeted, showing the importance of targeting energy at the end-organ for effective outcomes.Additionally, the movement of body organs, such as caused by RIM, adds to the complexity of maintaining US energy on the target.

[0055] We propose integrating US tracking algorithms for non-invasive US stimulation devices to prevent misalignment with the targeted tissue and to improve the effectiveness of the therapy. The challenges and innovations of the suggested framework are discussed in the following paragraphs. The methods and enhancements presented not only apply to end-organ US stimulation therapies but also hold relevance for a wide range of applications, including organ-targeted drug or surgical therapies and real-time imaging calibrations.

[0056] US imaging has the potential to offer superior temporal resolution in capturing dynamic organ movements coupled with cost-effectiveness, real-time capabilities, and safety compared to other imaging modalities that has increased its popularity in quantitative applications. Classification, localization, description, and motion estimation of tools or structures are examples of these applications, where there is a push to incorporate automation into them. However, the majority of available datasets for organs’ motion have relied on CT, MRI, and X-ray examinations, and there is a limited number of studies dedicated to using US images for model development. From an anatomical perspective, RIM models have primarily been suggested for uses in the liver, heart, and lungs. Most of them were developed for a specific task such as radiotherapy, minimally invasive cardiac interventions, and ablation of liver tumors. However, motion estimation of the spleen has not received enough focus in theUMN 2025-187-02Quarles 920171.00690 literature, and there is now increasing interest in spleen tracking due to recent discoveries for potentially treating a range of health disorders using US splenic stimulation. The potential for splenic US to treat various conditions includes rheumatoid arthritis, inflammatory bowel disease, acute kidney injury, myocarditis, pulmonary hypertension, pancreatitis, sepsis, and inflammatory-based brain disorders; in addition to opportunities with liver US stimulation for diabetes and metabolic disorders.

[0057] Beyond just the spleen and liver, diverse approaches for motion registration and tracking methods of different body regions have been thoroughly investigated, and there is a growing trend to integrate deep learning techniques into the optimization components of the registration methods. However, most of the investigated deep learning models require large, diverse training datasets for optimal model convergence. In various computer vision applications, large datasets are accessible for developing estimation and prediction models. However, in the realm of medical imaging, such datasets are often lacking. Even when medical images are accessible, creating high-quality models is challenging due to the complexity and cost of precise annotation. To avoid this problem, an alternate approach would be to implement methods that minimize or eliminate the annotation process.

[0058] Structures on US sequences of the abdomen have been tracked utilizing speckle tracking using block-matching, feature-based approaches, optical flow techniques, template matching, and active contours. However, US tracking poses challenges due to noise and speckle patterns, affecting block-matching accuracy. Feature-based approaches face reliability issues with low contrast and noise in US images. Optical flow calculations are less robust due to variations in tissue characteristics and noise. The variability in organ shapes hinders template-based tracking. Noisy images and the need for precise initialization impact the robustness of contour evolution in US tracking. To address these challenges and enhance the accuracy of organ tracking in US, we propose an improved version of the Kanade-Lucas-Tomasi (KLT) tracker. The KLT tracker excels in feature tracking and motion estimation, making it particularly well-suited for overcoming the challenges posed by traditional tracking methods in US domain.

[0059] We have proposed an automatic, fast (operates in real-time speed) and robust motion estimation technique that combines a proposed enhanced KLT feature tracker (EKLT) with a long short-term memory (LSTM) prediction network. This is a novel network architecture that utilizes the strengths of both LSTM and KLT algorithms. The KLT block offers the ability to achieve subpixel accuracy and eliminates the necessity for an annotatedUMN 2025-187-02Quarles 920171.00690 dataset. The LSTM algorithm provides the ability to learn temporal relationships of location sequences that are both useful for location prediction and region of interest (ROI) adjustment.

[0060] Many variants and extensions of KLT tracker, such as accounting for pixels object / background likelihood, using a scale-space representation of the images, and learning linear models that predict geometric displacement as a function of appearance, are described in the literature. However, to the best of our knowledge this method has not been extended for medical US tracking. We have proposed an enhanced multi-ROI KLT (MR-EKLT) tracker to extract the RIM of the spleen and liver from 2-D US images. The robustness of this method has also been improved by using forward-backward error to detect tracking failures, as well as by employing pyramid images and a final selection strategy. It is common practice to update detected features every M frame in iterative KLT trackers. Nevertheless, using this method might be problematic if noisy frames or occlusion are present in subsequent frames. Our proposed LSTM module can address this challenge in US imaging by predicting the next locations of detected features. A ROI processing step in the case of muti-ROI tracking, and a post-processing block to adapt the method to possible motion in the US probe position have been incorporated.

[0061] To the best of our knowledge, motion estimation and tracking methods have not yet been investigated under different breathing rates and rhythms, and their effects on the organ’s deformation rate. We have suggested the use of the parameter rigid organ error (ROE) as a measurement of deformation, and to show the efficiency and robustness of the proposed algorithm, as well as its performance evaluated on different breathing patterns (40 image sequences) including shallow versus deep breaths and fast versus slow respiration rates. These patterns induce different rates of deformation on the spleen and varying amplitudes of movement. The performance of our proposed EKLT-LSTM tracker has also been evaluated on liver motion to show its capability to track other organs. The proposed framework benefits from the advantages of both tracking models and the creation of reliable healthcare datasets.

[0062] This paper is organized as follows. Section II describes the materials used in this study, the US data collection protocol and corresponding datasets, and manual annotation procedures. Section III provides an overview of the general architecture of the proposed EKLT-LSTM tracking framework. Subsequently, in section IV, each module of the tracker, including the original KLT and feature detection, EKLT, and LSTM architectures will be illustrated in detail. Section V and VI provide post-processing steps and evaluation metricsUMN 2025-187-02Quarles 920171.00690 defined in the algorithms. A case study has been investigated in section VII. The results are provided in section VIII. Finally, the conclusion and perspectives are discussed in section IX.

[0063] II. Data Collection and Annotation

[0064] II.1 Spleen

[0065] This study was granted ethical approval by the University of Minnesota’s Institutional Review Board (IRB). Each participant provided informed, written consent prior to the data collection. This is an ongoing study, and volunteers are healthy individuals over the age of 18. At the time of developing the tracking algorithms, the motion of the spleen in 50 participants was recorded, and 20 of them were successfully recorded for all types of breathing. From these 20 participants, we selected those with diverse demographic data (age, gender, and BMI) to assess the algorithm’s performance across different demographic profiles. The demographic data of participants is shown in table I, which includes five males and five females in the range of 21 to 76 years old, and their BMI ranged from 18.8 (close to underweight as categorized by the world health organization) to 35.5 (obese).

[0066] The US data used in this study was acquired with a SonoSite Edge II US system from Fujifilm with an rC60xi curved linear transducer. The image resolution is 640 x 480 pixels with a pixel size of about 0.4 x 0.4 mm and temporal resolution of 7.5 frames per second. The 2-D US image sequences were obtained from ten healthy volunteers who were asked to vary their breathing frequency in the supine position. The transducer was oriented along space between the ribs (a diagonal axis to left flank) over the spleen. FIG. 1 shows the transducer placement and an example of the resulting image. In US images, the blue dot corresponds to the raised dot on the transducer wand. The participants were asked to breathe in four patterns: shallow-slow, shallow-fast, deep-slow, and deep-fast. At least ten cycles of breathing for each type were recorded. Each sequence contains a different number of frames ranging from 63 to 910 frames. Our study aims to develop a tracking algorithm for neuromodulation therapy, wherein US stimulation targets an organ to activate its cellular and neural pathways. This approach is intended to modulate multiple anti-inflammatory pathways within the spleen, in which previous studies in animals and humans have shown that ultrasound stimulation of different regions of the spleen can equivalently reduce cytokine production across the body. Furthermore, this previous research also showed that cutting the vagus nerve in the cervical neck area greatly eliminates the anti-inflammatory effect when applying US to the spleen, confirming that the functional effect involves a reflex pathway up to the brain and back down the vagus nerve to the body and is not simply a localized USUMN 2025-187-02Quarles 920171.00690 activation effect within the spleen. Therefore, these findings support that different types of cells and pathways are involved with the anti-inflammatory pathway, also supported by multiple studies showing that US is not able to readily excite nerves alone with low intensities currently used for clinical applications. In the spleen, there are different regions that can be targeted but one convenient target is in the hilum region consisting of a high density of vasculature and nerves that allow better visualization in imaging modalities.Consequently, our ROIs are defined around these major vessels; the advantage of using these structures is that vascular structures are well-visualized and prominent in US images, as well as those structures corresponding to approximately the center or mid-region of the spleen. The spleen dataset was collected such that the hilum is visible in all frames. We aim to target the nerve, splenocytes, and immune cells in close proximity to this hilum region to ensure effective and precise targeting, while also maintaining safety by avoiding unintended stimulation of other organs. After collecting US data, a set of points were manually marked on the spleen sonographs. For this manual annotation, we first examine all frames to identify which landmarks on hilum are visible and trackable across the frames. Once identified, we define an ROI in the first frame, around the anatomical landmark, specifically a vessel structure in the hilum (this ROI is also used later for the tracker as its input). Within this region, we select 3 to 4 visible and trackable points (pixels) for tracking. For sequences with slow breathing in spleen tracking, manual annotations were performed every fourth frame, corresponding to a time resolution of 0.25 seconds per annotation. This sampling rate was chosen to adequately capture the spleen’s motion during breathing cycles. For sequences with fast breathing, annotations were made every other frame, resulting in a time resolution of 0.5 seconds per annotation. This increased sampling frequency was used to ensure that the more rapid motion was accurately captured. We then obtain the motion trajectory for these points, and if there is any inconsistency in the motion for a specific frame between the points, we review that frame again and use information from the previous or next frame to improve accuracy. We have also considered another ROI in the hilum and applied the same process to increase the consistency of the annotation. Manual annotations were performed using MATLAB’s annotation tool, and to provide more accuracy two annotators have performed this process. Using this technique, the selected points trajectories in the x and y directions were calculated from images to yield continuous displacement-time plots for at least ten respiratory cycles as a manually annotated dataset. We have calculated their average toUMN 2025-187-02Quarles 920171.00690 represent the general motion of each ROI. Several examples of manual annotations for the spleen on the first frame of sequences are shown in FIG. 2.

[0067] II.2 Liver

[0068] We employed data from the Liver Ultrasound Tracking Challenge (CLUST), which was part of the Medical Image Computing and Computer Assisted Interventions 2014 Challenge. The CLUST challenge included 23 2-D ultrasound sequences of the liver, recorded from volunteer test subjects demonstrating free-breathing patterns. Notably, motion data for the liver of nine subjects can be accessed through citations to the relevant publications

[0062] ,

[0074] . The resulting 2650 to 14516 frames had a temporal and spatial resolution of 14-25 Hz and 0.3-0.7 mm, respectively. Since we utilized the 2-D liver dataset from the CLUST, and we did not have control over data collection (for spleen we focused on the hilum for data collection), we focused on the visibility of landmarks, specifically vessels, within the dataset. The defined annotation for this dataset was the center of the vessels, but given that our approach is optimized for edge, comer detection and regions with uneven intensities, such as vessel borders, we concentrated on these features for our liver dataset annotations. For all liver sequences, annotations were performed every fourth frame, consistent with the slow and normal breathing observed in the dataset. Manual tracking was conducted in a manner like spleen annotation. As mentioned in the Introduction section, there are few publicly available medical datasets related to organ motion in US imaging, with one of the most well-known being the CLUST dataset. We chose to use this dataset for two reasons. First, it allowed us to test our approach on a dataset that was not specifically developed for it, demonstrating its generalizability. Second, it enabled us to compare our method with previous approaches developed and suggested for this dataset, particularly in the context of organ motion, where there is a notable lack of research focused on the spleen. An example of a 2-D US image of the liver from this dataset, along with an annotated point, is depicted in FIG. 3.

[0069] III. EKLT-LSTM Tracking Framework

[0070] In this section, we present the overall tracking model developed in this study. The subsequent sections will delve into a detailed examination of its individual components.

[0071] FIG. 4 shows the overall tracking framework. In the initial frame of the sequence, the ROI is manually selected (ROI definition in the figure). This selected ROI is then inputted into the feature detection component, where points (or features) within the ROI containing texture information that are suitable for tracking are identified using the gradientUMN 2025-187-02Quarles 920171.00690 distribution of the structure tensor, which represents the gradient’s distribution in a specific region surrounding a point (more details in section IV). The selected features have been shown as detected features in FIG. 4. Subsequently, these features are tracked using an enhanced version of the KLT tracker, which is well-established for tracking good features. KLT has been improved for higher accuracy, and specific conditions for confident tracking have been established, which are presented in the next section. If these conditions are met throughout the sequence until the last frame, the tracker provides the final trajectory.However, in cases where the conditions are not satisfied, which occur in high noise and occlusion, the LSTM predictor developed in this study comes into play. The LSTM forecasts the ROI’s next location in the subsequent n frames. In this way, the tracker can jump from a noisy or an occlusion situation to an improved one, depending on the capability of the LSTM network. Another interesting outcome of using LSTM is that the LSTM can adjust the location of the ROI to a better location that contains more favorable features for tracking. This adjustment is accomplished by feeding the best features for tracking in the last frame with the satisfied condition to the LSTM, which will go on to predict the next ROIs with a center among these features. Finally, the nth predicted ROI is inputted into the feature detector, and this process will be repeated to reach the last frame and final motion trajectory.

[0072] In summary, two frameworks are proposed and adopted: the first utilizes the EKLT tracker, and the second combines the EKLT tracker with LSTM. These methods were developed based on the collected datasets. During data collection, participants are instructed to remain still, and the probe is kept in a consistent position across different breathing types. However, based on the differences in body structure and types of breathing, we have collected data with different qualities, which we categorize it into three groups: Group 1) Data where the ROI is visible throughout all frames, such as during slow and shallow breaths for spleen tracking and for all cases in the liver dataset. This group is tracked effectively and accurately using our suggested EKLT tracker. Group 2) Data where the ROI may be temporarily obscured, such as during deep breathing when the spleen moves off-screen, is shadowed by ribs, or is occluded by the lung during inhalation. Despite these short-term occlusions, the organ remains visible most of the time. For this group, we use an LSTM network to handle these short interruptions. We have established criteria based on breathing type and device sampling frequency, for example, if the organ is not visible for 1 or 2 seconds (in which the LSTM network is designed for it). The LSTM can be designed for longer times if more data is available. This approach allows us to utilize data that mightUMN 2025-187-02Quarles 920171.00690 otherwise be deemed unusable in other studies. Group 3) Data affected by prolonged occlusions due to rib shadowing, lung occlusion at the end of inhalation, or challenging body structures that complicate imaging. Deep breathing exacerbates these issues, leading to poor data quality. This group is excluded from our analysis. This is because there are no identifiable points for tracking when the organ is not visible, which would result in unreliable data.

[0073] IV. EKLT-LSTM Tracker Modules

[0074] In this section, we investigate each component of the EKLT-LSTM framework. First, we provide a brief overview of the KLT tracker’s foundational principles. We then delve into the enhancements we proposed for tracking US images. At the end of this section, the LSTM structure is discussed.

[0075] IV.1 KLT Tracking

[0076] Optical flow is a method for estimating the motion of pixels between consecutive frames in a video or image sequence. The fundamental assumption in optical flow is that pixel intensities do not change between consecutive frames as objects move. In the Introduction section, it was noted that optical flow can struggle in US imaging due to variations in tissue characteristics and noise. However optical flow can be combined with other motion estimation methods such as feature-based tracking to overcome these challenges. The KLT tracker provides this hybrid method. The main idea of the KLT algorithm is to minimize the sum of squared differences (SSD) between the pixel values in a local neighborhood surrounding a feature in the previous frame and the matching region in the current frame. In this process, the eigenvalues of the structure tensor, which are determined by the gradient of the image, are evaluated.

[0077] The given grayscale 2-D image is defined by I and a ROI in the image I by R. A point in R = (u, v) has changed by (α, β) in the next image at time t + τ. This means that by appropriately moving each point in the current image (or fixed image), a later image (sometimes referred to as a moving image) can be obtained:rn [UnU- / woij+ w' *’r)'v + v'f-'T)) = )

[0079] The amount of motion d = (α, β) is called the displacement vector of the point at R = (u, v). Selecting different points in the ROI will result in different amounts of displacement. The variation or motion fields can be represented by rigid and affine transformations. Since affine transformation includes rigid, scaling, and shearing transformations, it has been used to define the motion field:UMN 2025-187-02Quarles 920171.00690d = DR + d0(2)where,

[0080]

[0081] D is a deformation matrix and do is the translation of the ROI’s center (vectors are denoted using bold-face font). The purpose of tracking is to find four parameters in the deformation matrix and two translational parameters of do to determine changes in R between two consecutive frames. In general, smaller feature windows are selected to reduce depth discontinuity issues, which involves using a pure translational model with a zero-deformation matrix

[0066] , simplifying the tracking process as (4).d = d0, d = (x, y) (4)

[0082]

[0083] Building on the concept of KLT, which aims to minimize the SSD between pixel values in the neighborhood of a point or feature in the previous and current frames, the displacement vector d is chosen to minimize the SSD. We refer to this as the residual error, which is defined by (5).e(d) — ^2 ^2 w(u, 4- x. v 4- y) ~ If(u, t’))2(5)

[0084] ’’

[0085] where If and Imare fixed and moving images, respectively, and w(u, v) is the type of window that slides over the ROI and is called the weighting function, w could be a Gaussian-like function to emphasize the central area of the window. The error e(d) is calculated by summing the squared differences across all pixels within the ROI, weighted by w(u,v), to measure how well the moving image aligns with the fixed image. By minimizing e(d), we identify the optimal displacement vector that best aligns the moving image with the fixed image. In a KLT tracker, to minimize e, the SSD is differentiated with respect to the unknown displacement vector parameters (x, y), and the result is set to zero \dl / dx, dl / dy\ = 0. Differentiation represents the image gradients in the x and y directions. Using Taylor expansion, the result is approximated with a linear system (e.g., when the displacement d is much smaller than the window size). Based on these approximations, image intensities in the translated window can be described in (6) as the intensities in the original window plus a residual component that is almost linearly correlated with the translation vector, and the unknown of this system is the displacement vector.Im(u + x, v + y) = If(u, v) + g.d (6)UMN 2025-187-02Quarles 920171.00690

[0087] Substituting (6) in (5), then differentiating it with respect to d, and setting the result equal to zero yields the following system (G), which is calculating the error in closed form.Gd0= e (7)

[0088]

[0089] The tracker estimates the displacement for each selected points in subsequent frame by evaluating multiple potential displacements (do) and selecting the one with the minimum error. For this purpose, the matrix G must be both above the image noise and well-conditioned

[0066] , Corresponding to this fact, G must have two large eigenvalues that confirm the existence of high texture information in the selected window

[0066] , To show this, we have rewritten equation (5) as (8).SSD ≈ ΣΣ w(u,v)(Ix(u,v)x + Iy(u,v)y)2(8)

[0090] « ’

[0091] where Ixand Iyare derivations of Ifin the x and y directions, respectively. Then (8) can be rewritten in matrix form:Ix(u, v)Iy(u, v)Ix(u, v)Iy(u, v) Iy(u,v)2' < L2> < LJy >1<' ■ T T ~ s>' < ~-~ Zi f;2> j

[0092]

[0093] Matrix S is the structure tensor, which explains the gradient’s distribution in a specific region surrounding a point and makes the information independent of the observing coordinates. Angled brackets in (10) denote averaging. Two small eigenvalues correspond to an approximately constant intensity profile within a window. When there is a large and a small eigenvalue, it correlates to a unidirectional texture pattern and in this case an edge is discovered. Corners, or any other pattern that can be consistently tracked can be represented by two large eigenvalues. The greatest eigenvalue of a window cannot be infinitely large because the intensity changes are constrained by the biggest allowable pixel value. In conclusion, we only accept the window and track it frame by frame if the smaller eigenvalue is greater than a predefined threshold (At).min(λ1, λ2) > λt(11)

[0094] UMN 2025-187-02Quarles 920171.00690

[0095] We have defined the parameter q (0 < q < 1) as a fraction of the maximum corner metric value in the image to determine (At). Algorithm 1 summarizes the core KLT algorithm used.Algorithm 1 Core KLT Algorithm.INPUTS: ROI, w (size of Gaussian filter), λt.1) Calculate the gradient of points (G) in the selected window ROI in the x and y directions.2) Create the S matrix (10).3) Calculate the minimum eigenvalues of points.4) Find the points that have minimum eigenvalue greater than λt.5) Choose points as interest points or features that have their minimum eigenvalue as a local maximum.6) Start searching the neighboring region of selected points in step 5 and for each selected point, estimate displacement between consecutive frames (the motion can be translation or affine and can be estimated using the Lucas-Kanade). 7) Store displacement of each point and update their position (I).8) Repeat 6 and 7 in all frames to return long trajectory for each point.OUTPUT: Trajectory of organ’s motion.

[0096] -

[0097] When selecting the ROI, the algorithm identifies the most relevant features within this area (steps 1-5). Within the matrix S, certain points may exhibit two eigenvalues surpassing the predefined threshold. However, in step 5, the algorithm prioritizes points with the highest eigenvalues. Subsequently, in step 6, the algorithm initiates to search the area around the selected points in the previous frame. This neighboring area can be defined by a block size, B = [b, b] where b is an odd integer. The extent of this surrounding region can be adjusted based on the observed speed of motion. B is often chosen to balance the need for tracking accuracy with computational efficiency. For example, when a participant is taking slow breaths, a block size like [21, 21] can be considered, whereas for faster breathing, larger blocks like [31, 31] may be more appropriate. This flexibility allows for real-time adjustment in applications where faster processing is required. However, for our specific application, we have set the block size to [31, 31] for all sequences, which has been suitable for our needs.

[0098] IV.2 Multi-ROI Enhanced KLT (MR-EKLT)UMN 2025-187-02Quarles 920171.00690

[0099] In this section, we offer critical enhancements and refinements to Algorithm 1 to meet the unique demands of organ tracking in US imaging.

[0100] IV.2.1 Feature Tracking Optimization

[0101] To ensure that the feature detector responds to changing conditions and maintains stable tracking, we need to define quality metrics and adjust them. One technique to increase the robustness of the feature tracking is incorporation of pyramid images within the tracker. Pyramid images are generated by progressively down sampling the original image to create a hierarchy of image resolutions and a multi-scale representation of the input image. By adjusting the number of levels, the performance and accuracy of the tracking can be optimized. A lower number of levels are appropriate for organs such as spleen and liver that usually do not undergo significant changes in scale or size between frames. Higher numbers can be used when the scale of the organ does not remain consistent during tracking. Higher levels offer robustness to scale variations but may require more computational resources. In some embodiments, the level is set to 3 for reducing computational load while still providing robustness to scale changes.

[0102] When the tracker is tracking the identified feature (or pixel) in an ROI, it must continuously locate the corresponding feature in both the current and subsequent frames. Within the matching region of the current frame, the feature with the lowest SSD is selected as the corresponding feature. This process is repeated for all features identified as good features within the initial ROI across all frames in the sequence. Consequently, we obtain a number of tracked features or detected trajectories. In practice, the number of features tracked in a KLT tracker can vary from a few to hundreds. A few features within the ROI can be less reliable due to potential factors such as noise, occlusion, or objects moving off-screen. On the other hand, a high number of features is computationally demanding and not all of them are reliable and stable. To provide a trade-off between the number of features and the computational efficiency, we have defined the parameter Nth as the number of features with high reliability for tracking, where reliability is determined by a Confidence Score (CS, 0 < CS < 1) assigned to each feature. CS in (12) is a function of SSD and is used to select features with the highest accuracy:zSSDi,£75^ — 1 > (12)[0X03] JUUJV5SD)

[0104] where Nf is the number of detected features in each frame. A higher CS indicates a stronger match between the two neighborhoods, with a perfect match having aUMN 2025-187-02Quarles 920171.00690 score of 1. Using this criterion, we consider only those features with a CS higher than a desired threshold (e.g., features with a score higher than 0.97) and we discard other features from the detected features in the current frame. The threshold is denoted by thscore in (13).

[0105] CSi> thscore, i = 1, 2, ..., Nf(13)

[0106] When the number of features meeting the conditions specified in equation (13), which we define as NCS, suddenly decreases in a frame, it indicates that this frame within the sequence presents certain challenges, such as potential noise or occlusions. To show this condition and follow reliable tracking, the tracker will continue to track the next frame only if the condition in (14) for that frame is satisfied. This condition is considered as confidence condition.(14)

[0107]

[0108] The KLT tracker only tracks a set of feature points rather than the entire image or ROI, making it computationally efficient compared to full-image or full-ROI tracking methods. In this study, we incorporated a confidence condition criterion, which the tracker evaluates alongside other improvements. This criterion reduces the number of features being tracked over time, maintaining at least Nth features. Additionally, by focusing on smaller image regions, we simplify the computation of displacement vectors. In such cases, the pyramidal KLT can effectively manage larger motions without compromising speed.

[0109] In cases where the goal of tracking is to provide motion data (such as for developing a correlation model with a device or sensor that can be used during the treatment phase for image-guided therapies), we automate the final step of trajectory selection using the forward-backward error

[0073] , In the forward-backward error method, the point is tracked forward in time by a tracker, which creates a trajectory. Then, by tracking backward in time from the last frame to the first, the validation trajectory is found. Finally, the two trajectories are compared, and the forward trajectory is considered incorrect if there is a substantial difference. We defined parameter P to show this difference in pixel. When the error exceeds the value specified for this attribute (P), the trajectory and associated point are deemed invalid.

[0110] When demonstrating the trajectory of a chosen ROI and plotting the motion trajectory of the spleen and liver (or other organ) based on motion of the selected ROI, we select the points from a closely spaced group of consecutive points that meet all mentioned accuracy and confidence conditions. This is preferable as isolated points may be moreUMN 2025-187-02Quarles 920171.00690 susceptible to noise. To compare with manual tracking, we select points that either align with the annotated points or have the minimum distance from them. The average of these points represents the overall motion of the ROI. All of these techniques significantly enhance the accuracy of the tracker and provided motion data.

[0111] In a typical KLT tracker, when the tracker loses features in a frame, new features are manually or automatically defined. It is common practice to update or re-detect features every M frames in iterative KLT feature trackers. This adjustment step is done to adjust to scene changes, handle lost features, or correct tracking issues. Nevertheless, the presence of noisy frames can pose a challenge when using this method. To enhance the accuracy of the tracker, we recommend combining it with a confidence condition. In such cases, increasing the value of M frames can be considered. Despite this improvement, during automatic reinitialization or feature redefinition, the last frame is used, which may not be effective when noise or occlusion affects multiple consecutive frames, highlighting the need for a prediction algorithm that we propose in the next section (section C).

[0112] IV.2.2 Multi-ROI Tracker

[0113] The algorithm is also upgraded by performing it on multiple ROIs to have a better estimation of the organ’s motion. Multiple ROIs can help to compare the displacement of different parts of an organ. It can also be used in other applications for tracking multiple organs. In our application, which is targeting a portion of the spleen (or liver), multi-ROI tracker is also helpful when one ROI is occluded or becomes noisy, so the tracker can track the other more reliable ROIs. In the case of integrating multiple ROIs, or in the case of removing discontinuities of motion trajectories resulting from detecting new features in a window when (14) is not satisfied, we have proposed to only consider the variations of the ROIs and exert these variations on the selected ROI. Thus, at first, we can subtract the location of the point at the first frame of the ROI from all trajectory coordinates’ (e.g., normalize it) and exert these changes on the selected ROI. In this case, there is a gap between connecting points. To remove this gap, we use the last displacement (between the last two frames) of the previous ROI as an estimation for the displacement (A ) between the last frame of the previous ROI and first frame of the current ROI to fill the gap. The process has been shown in (15) assuming the displacement of the first ROI (Rl) is the interested ROI.d(Ri1) = RN- RN, i = 2, ..., nRnew(i) = RN- d(Ri1+ Δf) (15)R1= ΣRnew(i)

[0114] UMN 2025-187-02Quarles 920171.00690

[0115] d(Rii) is the difference between ith ROI (R ‘) and interested ROI (first ROI (RN1J). Algorithm 2 summarizes the steps of the MR-EKLT tracker.Algorithm 2 Enhanced KLT Algorithm.INPUTS: ROIs, w (size of Gaussian filter), λt, thscore, Nth, P, M, Ntotal(number of total frames in a sequence).1) Start algorithm and n = 1.While (n < Ntotal){2) Read the first frame of the next M frames and count the frames in parameter c.3) Define m as the number of ROIs on the frame.While (in each frame NCS≥ Nth^s > and c < Af)4) Go to steps 1-7 of algorithm 1 for all ROIs.5) Store (CSi) of each point along with their displacement and location and c = c + 1.6) Repeat 4 and 5 in all frames to return long trajectory for each point.}7) Update n (as the last qualified frame for tracking).8) Forward-backward error for all trajectories and final point selection strategy.9) ROI processing.OUTPUT location and displacements of points in multiple ROIs..

[0116]

[0117] An example case of the tracking process when the parameters were set as follows: w = 5, λ(q = 0.05), thscore = 0.98, Nth = 5, and P = 3 is shown in FIG. 5. The values for w, q, and P have been selected empirically and are widely accepted, thscore and Nth are selected to achieve high accuracy of tracking by EKLT. In FIG. 5, the first column shows two selected ROIs in red and blue color on participant 4 in a supine position, in which the participant was taking slow and shallow (normal) breaths. The second column shows the detected features at the first frame, and the third column shows trackable points, which met the confidence condition and remained as trackable points after 10 cycles of breathing. Using the MR-EKLT tracker, the spleen’s frame by frame motion was acquired and compared with manual measurements that resulted in 0.2mm error with 0.2mm standard deviation.UMN 2025-187-02Quarles 920171.00690

[0118] The process for liver tracking with the same parameter setting for tracking algorithm on volunteer 2 of the liver dataset, is illustrated in FIG. 6. The first column displays two selected ROIs in red and blue colors on the liver in a supine position, while the participant is taking normal breaths. The second column in FIG. 6 shows the detected features in the first frame, and the third column shows the trackable points that met the confidence condition and remained trackable after 5 minutes and 28 seconds. The tracker is capable of accurately following the ROIs (features) with an error margin of 0.1mm. The MR-EKLT was able to track all 9 liver motion sequences [62, 74] and its mean errors on this dataset was about 0.3mm.

[0119] IV.3 LSTM Predictor

[0120] Despite all the improvements made in Algorithm 2, in the presence of noise or occlusion in consecutive frames, tracking iterations may frequently occur, as discussed in previous section. To address this challenge, we suggest a new tracking architecture that integrates an LSTM network to predict the next location of the ROI in the next r time steps of the sequence. In this way, the LSTM can provide a more reliable ROI for the feature detection module and subsequently for the EKLT tracker in scenarios of occlusion or noise until reaching a confident situation. This approach is utilized for the data categorized in Group 2, as introduced in Section III. Examples of this group are shown in FIG. 7. In this figure, the first row (case (a)) displays five consecutive frames from a US sequence during deep breathing. In the third frame, the spleen begins to be occluded by the lung during inhalation, causing the tracker to lose the red ROI, which was defined on the hilum, for two consecutive frames. After these two occluded frames, the red ROI becomes visible again. However, the blue ROI, located closer to the caudal point of the spleen, remains visible throughout the sequence. In the second example (case (b)), showing nine consecutive frames from another participant in deep breathing, the red ROI is lost for five frames, from frame 3 to frame 7, but becomes visible again in frame 8. To handle this common type of occlusion in US imaging, we propose using the LSTM network to bridge these occlusions. The red ROIs in frame 5 of case (a) and frame 8 of case (b) represent the predicted ROIs generated by the proposed LSTM network. The details of this network are described as follows.

[0121] LSTM networks are a type of recurrent neural network (RNN) specifically designed to model and predict sequential data. In general, an LSTM network receives input sequences where each element of the sequence consists of multiple features. These inputs are fed into the network one time step at a time, allowing the LSTM to capture temporalUMN 2025-187-02Quarles 920171.00690 dependencies and patterns within the data. An LSTM network consists of memory cells that have the capacity to predict futures based on input data, update their states over time, and store information for an extended period. The output of an LSTM can vary depending on the configuration but typically involves predicting future values based on the learned patterns from the input sequences.

[0122] FIG. 5 depicts the architecture of our prediction model, which is designed for accurately predicting spleen location. The input layer processes sequential data, and it is configured to match the dimensionality of the input time series data. The input to the LSTM layer consists of five sequential samples representing the current and the four preceding time steps, denoted as m(t), m(t - 1),..., m(t - 4). These inputs are fed into the LSTM layer one at a time in a parallel format, allowing the LSTM to learn from the temporal dynamics of the breathing patterns. The LSTM is trained to predict the next T locations in the sequence, m(t+l), m(t+2),..., m(t+r ), effectively forecasting future breathing positions based on the recent history.

[0123] We employed the Adam optimizer for training the LSTM network, which is an adaptive learning rate optimization algorithm that combines the advantages of Adaptive Gradient Algorithm (AdaGrad) and Root Mean Square Propagation (RMSProp). We have conducted an in-depth exploration of various LSTM network architectures, including models with a single LSTM layer and stacked LSTM layers. We found that adding extra layers did not statistically significantly improve prediction accuracy. Consequently, we opted to employ a single LSTM layer in our final architecture. We also investigated the impact of increasing the number of hidden LSTM units within the LSTM layer. We observed improved accuracy when we added 60 to 100 hidden units, but further increases did not significantly reduce prediction error. Therefore, the number of hidden LSTM units was set to 100. It’s important to note that adding too many hidden units can lead to overfitting. The learning rate was also varied between 0.1 and 0.001 and was set to 0.01. Additionally, the selection of five time steps for input and ten for output is based on the dataset categorized as Group 2 in Section III. Based on the normal breathing frequency of 12 to 20 breaths per minute, the average duration of one breathing cycle is around 4 seconds. Given the sampling rate of the US imaging device at 7.5 frames per second, 10 to 15 frames correspond to approximately 1.3 to 2 seconds, represents a portion of one breathing cycle. This was the rationale for starting with these values for prediction. However, since the data was collected at different breathing rates and subjects may breathe differently, this number may represent more than just a portion of aUMN 2025-187-02Quarles 920171.00690 single cycle — it could encompass one or more full cycles. We have provided information on each participant’s breathing rate during various breathing phases, along with other related results, which can give a sense of how fast the breathing is. Additionally, it should be noted that to train the LSTM model to predict further time steps ahead, more data is required. In this study the T was set to 10 frames ahead.

[0124] Following the LSTM layer, a fully connected layer aggregates the LSTM layer’s outputs and prepares them for the final prediction. The regression layer is positioned at the output and responsible for producing continuous predictions. It helps us forecast future values in the time series. The prediction network can predict the next T time steps of the sequence.

[0125] We implemented walk-forward cross-validation technique to ensure effective training and validation of the LSTM network. This method helps prevent data leakage by ensuring that the model is only trained on past data and validated on future data, which is crucial for time series applications. We defined an initial fold of data used for training the model. This window encompasses a specific number (w train) of consecutive samples.Following the training window, a subsequent window (wtest) of data is designated for testing. The network is trained over wtrainto predict the next τ locations and is tested on wtest. The predictions in test window are compared against the actual values using RMSE. After each training iteration, the training window is incrementally expanded with steptrain samples and is tested on next wtest samples. This validation process ensures that the model continually learns from all available past information without incorporating future data, thus maintaining the integrity of the temporal sequence. This approach also ensures that different segments of the data, potentially exhibiting varying patterns, are used for testing, providing a comprehensive assessment of the model’s predictive capabilities.

[0126] The walk-forward process is repeated across the entire dataset, excluding the last wtest + gap samples, which are reserved as unseen data for the final evaluation (as application phase). This gap acts as a buffer, ensuring that the model does not inadvertently learn from data points that are too close in time to each other.

[0127] It should be noted that for each participant and each of the four breathing types, the LSTM network is trained separately on the corresponding dataset and for the part of the data with good quality (less noise and no occlusion). This allows the model to specialize in the specific characteristics of each breathing pattern. During instances of temporary occlusions or when the organ is off-screen — particularly during deep breaths — theUMN 2025-187-02Quarles 920171.00690 trained network is utilized to predict the missing frames based on the available data. The results provided in table II are results of trained, validated, and tested network using tram = 150, steptrain = 50, Wtest = 40, and gap = 10 for validation over slow-breathing dataset and using wtrain= 50, steptrain= 20, wtest= 20, and gap = 5 for validation over fast-breathing dataset to assess the LSTM performance. In this table, the third column shows the average validation error. For instance, in the case of slow breathing, the average validation error was calculated over 5 windows, each containing 40 samples, when the model was trained on 150, 200, 250, 300, and 350 samples. The average value and standard deviation of these errors are presented in this column. The last column shows the test results on unseen data, which is separated from the trained data by a distance determined by the parameter gap. The mean validation and test errors across all subjects were 0.74±0.24mm and 0.76mm, respectively. After the LSTM model is trained, it is integrated into the EKLT tracker to prevent it from stopping when occlusion occurs. In that case, the last 5 locations of the ROI in last 5 frames (before occlusion happened) are inputted to the LSTM and the T th predicted location provided by LSTM is considered as the center of ROI for new ROI definition.UMN 2025-187-02Quarles 920171.00690 Subject Respiration Avg validation! Test err®error (nun)! (nun) S\ w Mcil l> *w ' i $6 i 9 i i iLfe1TW Slow Shallow f > 49 I 029 w Slow Deep J (4 e- u ] w im TasTBSTRw ' "i 2? th U 02 rast Deep 1 M t o io orS\-w Shalh«w DiS i (s 85 v Deep 1 M tr 9 _ 1 1 9.35Ta't SLLW * 1 ± ~ 6 / n. [W; Rui Drop ' > w> •; o.i 7 jgZZZZj Mov ShdkwsfFbO 1 0s19**'w Sw*l*o***w*****>vs D*^**ro**'vhyhp********KKKKKK* u.49 U: 9 2o 9.20:Hst Sha'hw O4ih% ffi. BTao Drop L0O o.2<j [0.555 S\-w Shalbwy 0.50:1 9.17 j 0.4?Slow Deep O.t. M ± 9.18 | 08Fast Mn low 1177 ± 9.41 | 0 SIFa't Deep oO O. M Lil6Mew Drop:9 57 i 0 1 “ [ 9.49hex? Sha'hjw J (s9 -t* 9 pj 9.791 axt Drop 'i e> t 9 25 [ 9.631. S3.ro. Mauhro • < 52 i 9. U j 9.51SW Deep 1 99 t p 31 j 2J0Past Sh flitiS 9 06 T 0 J? pi71ba't Drop 5 0, - UJ 1 H 92X Siero Sh Jkws1L52 -r 0.1 I [ 1144Mow Deep 0.23 ± 9.25 1 0.520.07 ± 0.41 [ 9.66I j st Deep 0.38 ± 0.26 | 9.959 S\ w Shallow 1.45:t: 9.21 | 1.25S\ w Deep 0.58:1: 9.38 j 0.68bast Shallow 1 12 ± 9 M j 1.34Bist Drop _ pj219 Skw Shulk»w k-<> e 03^ 1 0*42Slow Drop ' LbO i 0 19 [ 1.56Fast Shallow 1?u — p t l [ LWFast DropS. MD ro 0.43 J (1.83

[0128]

[0129] Table II: LSTM prediction accuracy to estimate the location of the best feature for tracking in the next 10 frames of the sequence. The walk-forward cross validation parameters are wtrain = 150, steptrain = 50, wtest = 40, and gap = 10. For slow-breathing dataset and wtrain = 50, steptrain = 20, wtest = 20, and gap = 5 over fast-breathing dataset.

[0130] V. Post-ProcessingUMN 2025-187-02Quarles 920171.00690

[0131] In the tracking procedure, we have assumed that there is a negligible interframe motion of the US probe during imaging. As the liver dataset was obtained at a fixed location, the inter-frame motion situation was not an issue. However, in collecting the spleen dataset, involuntary hand movements during the imaging process might happen and can lead to undesired artifacts in the image, and it is required to eliminate these artifacts from the final data set. This step can be achieved through various techniques. We have used the least squares method to estimate the trend in data. This method involves finding the best-fit line or polynomial to the motion data. If we assume the motion data (spleen’s motion in x-axis at time t) is shown by mt and the best-fit polynomial is modeled as a p-order polynomial using, bt= b0+b1t+...+bptp, then the estimated polynomial trend can be defined as,(16)

[0132] bi

[0133] Complex trends and fluctuations in the data can be captured by higher order polynomials. However, using higher orders leads to capturing noise or random fluctuations in the data rather than the underlying actual trend. The detrend data can be obtained as,m̂t= mt- b̂0- b̂1t - ... - b̂ptp(17)

[0134]

[0135] where b̂0, b̂1, ..., b̂pdenote the corresponding least squares estimates.Normalizing and detrending the data ensures that the data is comparable and allows for meaningful analysis. The resulting trajectories from FIG. 5 for two ROIs and detrended and normalized motion trajectories when p = 4, are shown in FIG. 9. It should be noted that this movement is not observed in all participants but is noted in some instances.

[0136] VI. Performance Metrics

[0137] VI.1 Accuracy and Error

[0138] To evaluate the accuracy of the resulting trajectory from the automatic algorithm, it can be compared to manual motion representation. For this purpose, only the frames that were manually annotated were used to evaluate the tracking algorithm’s performance. After selecting those frames, Since the defined ROIs in the first frame are the same for both manual and automatic tracking, we selected points in the first frame that either align with the manual annotations or are closest to them, based on the tracker’s ability to successfully follow them. The average of these points at each frame is then used to generate the trajectory of ROI motion for both the manual and automatic trackers. This results in mAUMN 2025-187-02Quarles 920171.00690 and mM representing the automatic and manual tracking data sequences, respectively, the error can be defined for each frame:

[0139] E(i) = ||mA(i) - mM(i)||

[0140] Where i is ithannotated frame and ||.|| represents the Euclidean distance between manual and estimated location of the ROI.

[0141] VI.2 Breathing Rate and Rigid Organ Error (ROE)

[0142] The error calculated in (18) might be affected by several factors, and it is not error-free. Some of these factors include the frequency of US devices, body structure, breathing rate, and deformation of the organ. We have used two parameters to represent the breathing rate and deformation of the organ in each image stream. The accuracy of each tracked frame sequence is presented along with these parameters to investigate the effect of them on the tracker and evaluate how accurate the tracker can perform in different speed of breathing and deformation.

[0143] The parameter, breaths per minute (BPM), has been defined as the rate of breathing that is correlated to how fast the spleen is moving in each video. To ensure about this correlation, we used a respiratory belt to obtain the respiratory signal. We manually tracked spleen motion and measured the number of troughs to peaks for both the respiratory signal and spleen motion signal. The correlation between these measurements was found to be 1. This indicates that the respiratory rate (breaths per minute) and the number of peaks (cycles) in the motion signal are identical. Thus, we count the number of peaks in the motion trajectory data to determine the respiration rate (breaths per minute) or the organ motion rate (motion cycles per minute).

[0144] There is no exact measure of how much deformation the KLT tracker can resist, as it depends on factors like the type of deformation, the quality of feature points, noise, image resolution, and implementation details. The proposed tracker performs well when the tracked ROI undergoes small to moderate changes in appearance, such as rotations, translations, or mild non-rigid deformations. However, it may struggle with large non-rigid changes. To account for this, the ROE parameter has been used in this study to quantify deformation and irregularities of the organ during respiration. To show the ROE of the organ, an ROI is selected and then the proposed tracker will track the region. The trajectory of qualified points on the region is extracted. The difference between each pair of points is calculated and saved during tracking. The mean ROE is calculated for each pair of points.UMN 2025-187-02Quarles 920171.00690 The maximum difference of pairs of points (or standard deviation of the pairs’ differences) is defined as the ROE of the region.

[0145] VII. Results and Discussion

[0146] In this section, we have assessed the performance of the suggested frameworks concerning the motion of the spleen and liver in 2D US imaging. The spleen and liver motion sequences in this study were recorded or used following the data collection protocol outlined in section II. Based on our observations of the dataset, we can distinguish two case studies determined by the quality of the recordings, which necessitate the use of either the EKLT algorithm or the combined LSTM-EKLT algorithm. The initial category pertains to recordings characterized by high-quality images (Group 1). In such instances, the EKLT model can accurately track motion without the need for new ROI redefinition. The second category deals with images that contain high levels of noise and / or occlusions for a few frames. In these cases, it is necessary to reacquire features or the ROI using the EKLT-LSTM model. The results for these two frameworks are presented in the following subsections (A and B). FIG. 10 illustrates the number of sequences that could be tracked for each of these categories.

[0147] VII.1 EKLT Tracker Evaluation

[0148] In this section, we have assessed the performance of the EKLT tracker on all images. Based on the results, it is evident that out of the 49 image sequences (comprising 40 for the spleen and 9 for the liver), 24 sequences could be tracked by EKLT in one-step with high accuracy, eliminating the necessity for any re-definition step. This set of 24 sequences encompass all liver sequences, all of the spleen dataset related to shallow and slow breathing (10 sequences)), and half of the shallow and fast breathing cases (5 sequences). The accuracy of the proposed EKLT tracker on these 24 image sequences is presented in this section. The tracking process was performed on all spleen image sequences. Throughout all EKLT implementations, the parameters were consistently set as follows: w = 5, λ(q = 0.05), thscore = 0.98, Nth = 5, and p = 3. The results are presented in Table II and Table III. The first column displays the subject’s ID, while the second column shows the mean frame-by-frame error between manual annotation and EKLT estimation for the sequence. The third and fourth columns illustrate BPM as the speed of breathing and ROE as the deformation criteria, respectively. Since the motion of the spleen has not been studied as extensively as other organs, the manually and automatically calculated average spleen motion is shown in columns 5 and 6. This displacement was calculated as the average distance from theUMN 2025-187-02Quarles 920171.00690 minimum to the peak of motion, occurring during the transition from exhale to inhale. This information is particularly helpful for comparing the motion of the spleen during different breathing types, as provided in Tables II, III, V, VI, and VII. In addition, it can demonstrate the impact of motion amplitude on the tracker’s performance.

[0149] The acceptable range of error is determined by several factors, including the image resolution, the level of accuracy required for the application (e.g., tracking blood flow or tumor movement), and the intended use of the tracking data (e.g., research, treatment planning, diagnosis). For tracking algorithms, a few pixels or millimeters is often regarded as a suitable range of errors. In our application, which is targeting a moving ROI near to hilum for US stimulation [18, 57], and while such high accuracy is not necessary, the tables demonstrate mean errors of 0.4 ± 0.45mm for slow and shallow breathing and 0.54 ± 0.4mm for fast and shallow breathing. Considering that the US image’s pixels are around 0.4mm x 0.4mm, these results demonstrate the remarkably high accuracy of our enhanced KLT tracker of between 1 and 2 pixels. The maximum movement in slow and shallow breathing can be observed in participants 5, 7, and 9, who have relatively higher errors compared to the other participants.

[0150] The MR-EKLT tracker, with the same parameter setting used for spleen tracking, was able to track all 9 liver motion sequences and its mean error over these sequences was about 0.3 ± 0.2mm indicating high accuracy compared to the results reported for this dataset in other studies

[0075] ,

[0151] VII.2 EKLT-LSTM Evaluation

[0152] This section presents the results of tracking with the EKLT-LSTM model for image sequences that could not be tracked in the previous section. The performance of the LSTM network on predicting the location of the ROI in the next 10 frames was provided in table II in subsection C. We have combined the trained networks with EKLT tracker. The accuracy of the proposed EKLT-LSTM tracking algorithm was evaluated on 25 US image sequences, which could not be tracked in the one-step method in the previous section. Noise and occlusion in these sequences pose challenges for the EKLT tracker. For example, the first cycle of breathing might be trackable, but the second cycle is occluded. The ROI might be visible in the third and fourth cycles, but the fifth cycle is noisy or occluded, requiring the ROI to be defined several times.

[0153] For each image sequence, the first frame was input to the feature detection and EKLT tracker modules where the following values for their setting were considered: w = 5,UMN 2025-187-02Quarles 920171.00690 size of the Gaussian filter, q = 0.05 (At = 0.05 x maximum comer metric value), thscore= 0.98, Nth = 5, and P = 3. The EKLT continues to track if the confidence conditions are met, otherwise it stops tracking. In this case, based on the accuracy and final selection strategy, the best feature from the last frame that successfully was tracked by EKLT is selected, and its location sequence is the input to the trained LSTM network for that participant. The LSTM can predict the location of the selected feature in the next 10 frames. An ROI with a center located at the 10thpredicted location is defined and fed into the feature detection module. This process is repeated until the last frame of the sequence. In this way the model can handle the occlusion situation efficiently. The accuracy of the method is considered by evaluating the error for available frames of the sequence using (18). The average error and standard deviation of this error for all available frames in the entire sequence are provided in this section. Two examples of this process were shown in FIG. 7 but another detailed example of ROI prediction by the LSTM network as an input for EKLT tracker is shown in FIGS. 11 A- 1 IE for participant 5 in deep and slow breathing. FIG. 11 A shows the first frame of the sequence and two defined ROIs in red and blue on this frame. The reliable features detected by feature detection module and tracked with EKLT until the first case of occlusion and noise happen are shown in FIG. 1 IB. The frame in which tracker missed the features is displayed in FIG. 11C, and FIG. 1 ID demonstrates the predicted locations for the best feature after 10 frames. Finally, in FIG. 1 IE defined ROIs with their center located on the predicted locations are shown.s E ± std BFM ROE Number Number of Avg 1 (mm) (mm) switehiag motion frames to LSTM trum) [ 1.1 ± 0.5 11.70 "24 god 9 28.59 7 04 ± 04.1730 1.0 632 3 1043 3 0.9 ± 04 09.24 1.4 845 6 26.54 4 0.6 ± 0.4 30.90 0.3 561 10.50 5 1.1 ± 0.7 06.60 2.6 759 7 38.39 6 0.2 ± 04 27.00 0.7 905 2 19.38 7 0.9 ± 04 98.80 0.6 886 3 2843 8 O.o ± 0.7 17.70 2.4 839 22.50 9 3.0 ± 0.4 06.20 1.1 728 11 47.6010 1.2 ± 0.6 12.30 2.2 725 6 22.60

[0154]

[0155] Table V: Accuracy of EKLT-LSTM motion tracking model in noisy and occlusion cases measured for available frames in slow and deep breathing.UMN 2025-187-02Quarles 920171.00690

[0156] Table V, Table VI, and Table VII demonstrate the EKLT-LSTM model to track the spleen in different respiration rhythm and deformation rates. In these tables, such as Table II and Table III, the second column displays the mean frame-by-frame error between manual annotation and EKLT estimation for the sequence. However, the difference with Tables II and III is that this error is calculated for available frames — frames in which the organ is visible and not occluded, off-screen, or very noisy. The third and fourth columns illustrate BPM as the speed of breathing and ROE as the deformation criteria, respectively. To provide a better understanding of the error and available frames, the total number of frames in each sequence is shown in column 5, and the number of times the algorithm has switched (Nswttch) to predict the 10thlocation is shown in the next column. For example, in Table V, for Subject 1, the total number of frames is 800, but due to noise or other situations, the ROI has been missed 9 times. Each time after a miss, the network predicted the 10thlocation, and this information was inputted to the tracker. As a result, 90 frames have not been included in the error calculation, and 710 frames were considered in the error calculation. The last column in these tables shows the averaged spleen motion.s E ± aU lililll liiliiii Number Number of iAfgilllli uf total inching oOoiiii frames to LSTM~389 ”7554 0.8 £ 0.5 75X1 13 146 06.17 5 0.4 ± D.2 83X1 (18 105 09.41 6 0.4 ± 0.3 61 Xi to 95 04.259 1.1 ± 0.6 60.8 2.1 117 4 08.52

[0157]

[0158] Table VI: Accuracy of EKLT=LSTM motion tracking model in noisy and occlusion cases measured for available frames in fast and shallow breathing.Illi iBiiiiii Number Number of |O||||||«f mud swluhine motion frames to LSTM4.8 ± 2.3 38.5 2.0 226 6 3O.2O 0.8 ± 9.9 75.0 0.5 251 4 05.10 1.2 ± 6.9 37.5 3.3 185 4 13.16 0.7 ± 0.5 64.0 1.1 285 4 08.30 2.2 ± 1.2 79.0 5.9 277 5 13.49 0.4 ± 0.5 56.0 1.0 195 3 08.96 0.9 ± 0.6 22.5 0.5 257 3 20.07 0.9 ± 0.8 62.0 1.6 293 4 08.72 1.3 ± 0.9 39.7 8.1 260 6 20.7010 0.5 ± 0.4 35.7 2.2 184 4 12.90

[0159] UMN 2025-187-02Quarles 920171.00690

[0160] Table VII: Accuracy of EKLT-LSTM motion tracking model in noisy and occlusion cases measured for available frames in fast and deep breathing.

[0161] VIII. Discussion

[0162] On average, the most reliable breathing for the tracking algorithm is slow and shallow breathing where the tracker’s greatest error was for participant 1 at about 0.9 mm with standard deviation of 0.4mm. All of this group could be tracked using EKLT with high accuracy of 0.4 ± 0.45mm. The tracker in fast and shallow breathing has performance close to slow and shallow breathing (average error of 0.54 ± 0.28mm for sequences tracked by EKLT, and average error of 0.72 ± 0.43mm for sequences tracked by EKLT-LSTM in Table VI). The maximum error for the EKLT-LSTM tracking algorithm is obtained for participant 1 in deep and fast breathing with an error of 4.8 ± 2.3mm; then participant 9 in slow and deep breathing, which is about 3.0 ± 0.4mm. The average errors for slow and deep breathing and for fast and deep breathing were 1.05 ± 0.53mm and 1.37 ± 0.9mm, respectively. We observed the highest error during deep and fast breathing. One possible explanation is that this type of respiration tends to deviate from a regular pattern, making it more difficult to predict due to its irregularity.

[0163] In general, the error is influenced by several factors, including deformation, feature point quality, noise, image resolution, and breathing speed. We investigated the effects of motion amplitude, ROE, and BPM on tracking accuracy. Our results show that error is primarily affected by the amplitude of displacement, followed by ROE. This trend is evident in the data: as the amplitude increases, the error rises as well. For instance, the maximum spleen displacement was observed in participant 9 during slow and deep breathing, measuring approximately 47.6mm. This participant also exhibited the second highest tracking error. While deformation can generally contribute to increased error, the algorithm is capable of handling moderate deformation. Thus, a direct correlation between increased deformation and error cannot be established, as other factors like displacement amplitude and the type of deformation play crucial roles. For example, participant 9 also exhibited the highest ROE, measuring 8.1mm during fast and deep breathing. Despite this, the error was only 1.3 ± 0.9mm, which is relatively low. Fast and deep breathing exhibited the highest average ROE of 2.62 ± 2.5mm. In contrast, the average deformation for slow and shallow breathing was 0.4 ± 0.4mm, representing the lowest average ROE among the four breathing types.

[0164] Regarding the comparison of our tracker with other methods, we conducted two evaluations. The first evaluates our EKLT tracker against other methods, while theUMN 2025-187-02Quarles 920171.00690 second focuses on the EKLT-LSTM method. The EKLT tracker is designed for tracking visible landmarks, a common approach in studies such as the CLUST dataset for liver tracking. Using this enhanced tracker, we achieved an error of 0.4 ± 0.4 mm on the spleen dataset and 0.3 ± 0.2 mm on the liver dataset. This level of accuracy exceeds the requirements for our neuromodulation application. In

[0076] , various tracking algorithms developed for CLUST were compared, with the fusion method achieving the highest accuracy at 0.92 ± 0.98 mm (for 2D liver sequences). Another comparison in

[0077] proposed a similarity measure-based block matching technique for 2D liver ultrasound. They reported that the accuracy of their method was comparable to or exceeded other 2D tracking methods with error of 0.72mm, where the next closest algorithm had a mean tracking error of 1.21 mm in

[0078] , Deep learning methods, such as Siamese networks

[0079] , have also been proposed for this dataset. The network reported an error of 1.34 ± 2.75mm, but despite the high accuracy, it did not result in a significant improvement. Based on these results, our proposed EKLT algorithm demonstrates superior accuracy. However, our results for the liver dataset are based on 9 subjects. As far as we know, the EKLT-LSTM method is the first to address landmarks that are temporarily occluded, and no other methods have been developed for this specific purpose. Nevertheless, our results for occluded landmarks demonstrate high accuracy, even compared to the tracking methods used for visible landmarks.

[0165] We suggested an integrated deep learning with a conventional method for tracking but exploring different methods for this integration is promising future direction for landmark tracking algorithms, particularly in the annotation step. Combining deep learning’s ability to learn complex patterns with the interpretability and efficiency of traditional techniques could automate and enhance the accuracy of annotation, reducing the need for manual input. Further work could focus on optimizing these algorithms for real-time clinical use and integrating them seamlessly into clinical workflows, which is essential for practical deployment in medical settings. The tracking algorithm described herein receives motion data that can be integrated into stimulation devices, such as correlating with respiration monitoring sensors for accurate stimulation. More research is needed to develop a real-time algorithm for stimulation; however, pseudo-real-time or offline methods are also impactful in creating sensors or other devices designed for stimulation. This is what we plan to pursue in the future. Another avenue for improvement is addressing increased organ deformation, where advanced algorithms or biomechanical models could better handle large-scaleUMN 2025-187-02Quarles 920171.00690 deformations. Additionally, developing Al-based annotation tools could significantly speed up the annotation process while improving accuracy and consistency.

[0166] IX. Conclusion

[0167] In this study, an end-to-end automatic motion tracking approach that can be used over long image sequences, and even occluded and noisy US images has been developed. The method is the combination of an enhanced KLT tracker and an LSTM predictor. The tracker has been implemented with multiple accuracy enhancement techniques, and LSTM can adjust the ROI selection and skip occlusion and noisy images by predicting good locations in future images. The algorithm was applied to estimate liver and spleen’s motion in different breathing patterns and deformation rates, which showed high accuracy. Our proposed approach offers the potential to reduce time and labeling costs. Moreover, it can be advantageous in scenarios involving the tracking of multiple organs or structures. The techniques and advancements are not limited to US stimulation therapies of the spleen and liver; they are also applicable to a diverse array of applications. These applications include organ-targeted stimulation therapies with wearable devices beyond the spleen and liver (e.g., intestine regions, kidney, pancreas, and heart), real-time imaging calibrations, and guidance for drug targeting or surgical planning (e.g., implants or lesions).

[0168] Illustrative Embodiments of Methods and Systems Described Herein

[0169] FIG. 12 shows an example process 1200 to track an organ. At step 1202, an image from a time series of images is received. The images may be collected by an ultrasound device. At step 1204, a region of interest (ROI) is selected and features within the ROI are identified. In some embodiments, the ROI is manually selected. In some embodiments, the ROI is automatically selected. At step 1506, identified features are tracked from the image to a subsequent image collected later in the time series. At step 1208, a confidence level of the tracking is determined. The confidence level is assessed to determine if the image is acceptable. If the confidence score is not acceptable, a trained network may be used to predict the ROI’s location in a set of subsequent images collected over a time range. After the time range, the process may resume tracking and determining the confidence score of the tracking. At step 1210, a trajectory of the organ is outputted.

[0170] In FIG. 13, an example 1300 of a system (e.g., a data processing system) for tracking motion of an organ in accordance with some embodiments of the disclosed subject matter is shown.

[0171] In some embodiments, computing device 1304 and / or server 1316 can be anyUMN 2025-187-02Quarles 920171.00690 suitable computing device or combination of devices, such as a desktop computer, a laptop computer, a smartphone, a tablet computer, a wearable computer, a server computer, a virtual machine being executed by a physical computing device, etc. As described herein, system 1300 can present information about the motion of the organ to a user (e.g., a subject, researcher, and / or a physician).

[0172] In some embodiments, communication network 1302 can be any suitable communication network or combination of communication networks. In some embodiments, communication network 1302 can be any suitable communication network or combination of communication networks. For example, communication network 1302 can include a Wi-Fi network (which can include one or more wireless routers, one or more switches, etc.), a peer-to-peer network (e.g., a Bluetooth network), a cellular network (e.g., a 4G network, a 5G network, etc., complying with any suitable standard, such as CDMA, GSM, LTE, LTE Advanced, WiMAX, etc.), a wired network, etc. In some embodiments, communication network 1302 can be a local area network, a wide area network, a public network (e.g., the Internet), a private or semi-private network (e.g., a corporate or university intranet), any other suitable type of network, or any suitable combination of networks. Communications links shown in FIG. 13 can each be any suitable communications link or combination of communications links, such as wired links, fiber optic links, Wi-Fi links, Bluetooth links, cellular links, etc.

[0173] FIG. 13 additionally shows an example of hardware that can be used to implement computing device 1304 and server 1316 in accordance with some embodiments of the disclosed subject matter. In some embodiments, computing device 1304 can be used to execute one or more set of instructions to track motion of an organ.

[0174] As shown in FIG. 13, computing device 1304 can include one or more hardware processor 1306, one or more displays 1308, one or more inputs 1310, one or more communications 1312, and / or memory 1314. In some embodiments, processor 1306 can be any suitable hardware processor or combination of processors, such as central processing unit, a graphics processing unit, etc. In some embodiments, display 1308 can include any suitable display devices, such as a computer monitor, a touchscreen, a television, etc. In some embodiments, inputs 1310 can include any suitable input device and / or sensors that can be used to receive user input, such as a keyboard, a mouse, a touchscreen, a microphone, etc.

[0175] In some embodiments, communication systems 1312 can include any suitable hardware, firmware, and / or software for communicating information over communicationUMN 2025-187-02Quarles 920171.00690 network 1302 and / or any other suitable communication networks. For example, communications systems 1312 can include one or more transceivers, one or more communication chips and / or chip sets, etc. In a more particular example, communications systems 1312 can include hardware, firmware and / or software that can be used to establish a Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, etc.

[0176] In some embodiments, memory 1314 can include any suitable storage device or devices that can be used to store instructions, values, etc., that can be used, for example, by processor 1306 to present content using display 1308, to communicate with server 1316 via communications system(s) 1312, etc.

[0177] Memory 1314 can include any suitable volatile memory, non-volatile memory, storage, or any suitable combination thereof. For example, memory 1314 can include RAM, ROM, EEPROM, one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, etc. In some embodiments, memory 1314 can have encoded thereon a computer program for controlling operation of computing device 1304. In such embodiments, processor 1306 can execute at least a portion of the computer program to present content (e.g., images, user interfaces, graphics, tables, etc.), receive content from server 1316, transmit information to server 1316, etc.

[0178] In some embodiments, server 1316 can include a processor 1318, a display 1320, one or more inputs 1322, one or more communications systems 1324, and / or memory 1326. In some embodiments, processor 1318 can be any suitable hardware processor or combination of processors, such as a central processing unit, a graphics processing unit, etc. In some embodiments, display 1320 can include any suitable display devices, such as a computer monitor, a touchscreen, a television, etc. In some embodiments, inputs 1322 can include any suitable input devices and / or sensors that can be used to receive user input, such as a keyboard, a mouse, a touchscreen, a microphone, etc.

[0179] In some embodiments, communications systems 1324 can include any suitable hardware, firmware, and / or software for communicating information over communication network 1302 and / or any other suitable communication networks. For example, communications systems 1324 can include one or more transceivers, one or more communication chips and / or chip sets, etc. In a more particular example, communications systems 1324 can include hardware, firmware and / or software that can be used to establish a Wi-Fi connection, a Bluetooth connection, a cellular connection, an Ethernet connection, etc.

[0180] In some embodiments, memory 1326 can include any suitable storage deviceUMN 2025-187-02Quarles 920171.00690 or devices that can be used to store instructions, values, etc., that can be used, for example, by processor 1318 to present content using display 1320, to communicate with one or more computing devices 1304, etc. Memory 1326 can include any suitable volatile memory, nonvolatile memory, storage, or any suitable combination thereof. For example, memory 1326 can include RAM, ROM, EEPROM, one or more flash drives, one or more hard disks, one or more solid state drives, one or more optical drives, etc. In some embodiments, memory 1326 can have encoded thereon a server program for controlling operation of server 1316. In such embodiments, processor 1318 can execute at least a portion of the server program to transmit information and / or content (e.g., results of a tissue identification and / or classification, a user interface, etc.) to one or more computing devices 1304, receive information and / or content from one or more computing devices 1304, receive instructions from one or more devices (e.g., a personal computer, a laptop computer, a tablet computer, a smartphone, etc.), etc.

[0181] In some embodiments, any suitable computer readable media can be used for storing instructions for performing the functions and / or processes described herein. For example, in some embodiments, computer readable media can be transitory or non-transitory. For example, non-transitory computer readable media can include media such as magnetic media (such as hard disks, floppy disks, etc.), optical media (such as compact discs, digital video discs, Blu-ray discs, etc.), semiconductor media (such as RAM, Flash memory, electrically programmable read only memory (EPROM), electrically erasable programmable read only memory (EEPROM), etc.), any suitable media that is not fleeting or devoid of any semblance of permanence during transmission, and / or any suitable tangible media. As another example, transitory computer readable media can include signals on networks, in wires, conductors, optical fibers, circuits, or any suitable media that is fleeting and devoid of any semblance of permanence during transmission, and / or any suitable intangible media.

[0182] Each of the following references is incorporated by reference in its entirety:

[0183] [1] Q. Wang et al., “Exploring Respiratory Motion Tracking through Electrical Impedance Tomography,” IEEE Trans Instrum Meas, vol. 70, 2021, doi:10.1109 / TIM.2021.3083892.

[0184] [2]P. M. Shakeel, M. A. Burhanuddin, and M. I. Desa, “Lung cancer detection from CT image using improved profuse clustering and deep learning instantaneously trained neural networks,” Measurement (Lond), vol. 145, pp. 702-712, Oct. 2019.UMN 2025-187-02Quarles 920171.00690

[0185] [3] M. C. Yip, D. G. Lowe, S. E. Salcudean, R. N. Rohling, and C. Y. Nguan, “Tissue tracking and registration for image-guided surgery,” IEEE Trans Med Imaging, vol.31, no. 11, pp. 2169-2182, 2012.

[0186] [4] T. Shepherd et al., “Comparative study with new accuracy metrics for target volume contouring in PET image guided radiation therapy,” IEEE Trans Med Imaging, vol. 31, no. 11, pp. 2006-2024, 2012.

[0187] [5] F. Ernst, R. Du' richen, A. Schlaefer, and A. Schweikard, “Evaluating and comparing algorithms for respiratory motion prediction,” Phys Med Biol, vol. 58, no. 11, pp.3911-3929, Jun. 2013.

[0188] [6] A. J'ohl et al., “Performance comparison of prediction filters for respiratory motion tracking in radiotherapy,” Med Phys, vol. 47, no. 2, pp. 643-650, Feb. 2020.

[0189] [7] L. Zhang et al., “Spatio-Temporal Convolutional LSTMs for Tumor Growth Prediction by Learning 4D Longitudinal Patient Data,” IEEE Trans Med Imaging, vol. 39, no. 4, pp. 1114-1126, Apr. 2020.

[0190] [8] X. Li, Y. H. Lee, S. Mikaiel, J. Simonelli, T. C. Tsao, and H. H. Wu, “Respiratory motion prediction using fusion-based multi-rate kalman filtering and real-time golden-angle radial mri,” IEEE Trans Biomed Eng, vol. 67, no. 6, pp. 1727-1738, Jun. 2020.

[0191] [9] A., et al Hatamizadeh, “Transformers for 3D Medical Image Segmentation,” in 2022 IEEE / CVF Winter Conference on Applications of Computer Vision (WACV) IEEE, 2022 IEEE / CVF Winter Conference on Applications of Computer Vision (WACV), 2022.

[0192]

[0010] F. Isensee, P. F. Jaeger, S. A. A. Kohl, J. Petersen, and K. H. Maier-Hein, “nnU-Net: a self-configuring method for deep learning-based biomedical image segmentation,” Nat Methods, vol. 18, no. 2, pp. 203-211, Feb. 2021.

[0193]

[0011] E. Mastella et al., “Validation of a pretreatment delivery quality assurance method for the CyberKnife Synchrony system,” Med Phys, vol. 43, no. 8, pp. 4565-4574, Aug. 2016.

[0194]

[0012] T. Depuydt et al., “Geometric accuracy of a novel gimbals based radiation therapy tumor tracking system,” Radiotherapy and Oncology, vol. 98, no. 3, pp. 365-372, Mar. 2011.

[0195]

[0013] A. Schweikard, H. Shiomi, and J. Adler, “Respiration tracking in radiosurgery,” Med Phys, vol. 31, no. 10, pp. 2738-2741, 2004.UMN 2025-187-02Quarles 920171.00690

[0196]

[0014] C. Cavedon, “Real-time control of respiratory motion: Beyond radiation therapy,” Physica Medica, vol. 66. Associazione Italiana di Fisica Medica, pp. 104-112, Oct.01, 2019.

[0197]

[0015] N. R. F. Huttinga, T. Bruijnen, C. A. T. van den Berg, and A. Sbrizzi, “Real-time non-rigid 3D respiratory motion estimation for MR-guided radiotherapy using MR-MOTUS,” IEEE Transactions on Medical Imaging 41.2, Apr. 2021.

[0198]

[0016] V. Cotero et al., “Noninvasive sub-organ ultrasound stimulation for targeted neuromodulation,” Nat Commun, vol. 10, no. 1, Dec. 2019.

[0199]

[0017] D. P. Zachs et al., “Noninvasive ultrasound stimulation of the spleen to treat inflammatory arthritis,” Nat Commun, vol. 10, no. 1, Dec. 2019.

[0200]

[0018] S. Zanos et al., “Focused ultrasound neuromodulation of the spleen activates an anti-inflammatory response in humans,” Brain Stimul, vol. 16, no. 3, pp. 703-711, May 2023, doi: 10.1016 / j.brs.2023.04.003.

[0201]

[0019] F. A. Koopman et al., “Vagus nerve stimulation inhibits cytokine production and attenuates disease severity in rheumatoid arthritis,” Proceedings of the National Academy of Sciences, vol. 113, no. 29, pp. 8284-8289, Jul. 2016, doi:10.1073 / pnas.1605635113.

[0202]

[0020] U. Ahmed et al., “Ultrasound Neuromodulation of the Spleen Has Time- Dependent Anti-Inflammatory Effect in a Pneumonia Model,” Front Immunol, vol. 13, Jun.2022.

[0203]

[0021] D. Martelli, M. J. McKinley, and R. M. McAllen, “The cholinergic antiinflammatory pathway: A critical review,” Auton Neurosci, vol. 182, pp. 65-69, 2014.

[0204]

[0022] V. Cotero et al., “Stimulation of the hepatoportal nerve plexus with focused ultrasound restores glucose homoeostasis in diabetic mice, rats and swine.,” Nat Biomed Eng, vol. 6, no. 6, pp. 683-705, Jun. 2022, doi: 10.1038 / s41551- 022-00870-w.

[0205]

[0023] J. Ashe et al., “Investigation of liver-targeted peripheral focused ultrasound stimulation (pFUS) and its effect on glucose homeostasis and insulin resistance in type 2 diabetes mellitus: a proof of concept, phase 1 trial.,” QJM, vol. 116, no. 8, pp. 667-685, Sep. 2023, doi: 10.1093 / qjmed / hcad098.

[0206]

[0024] E. Harris, D. Fontanarosa, and C. Baldock, “In the future, ultrasound guidance in radiotherapy will become a clinical standard,” Physical and Engineering Sciences in Medicine, vol. 44, no. 2. Springer Science and Business Media Deutschland GmbH, pp. 347-350, Jun. 01, 2021.UMN 2025-187-02Quarles 920171.00690

[0207]

[0025] Esther G. C. Troost, “Image-Guided High-Precision Radiotherapy”. Springer International Publishing, 2022.

[0208]

[0026] A.-R. Ali, A. E. Samir, and P. Guo, “Self-Supervised Learning for Accurate Liver View Classification in Ultrasound Images with Minimal Labeled Data,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition., 2023.

[0209]

[0027] K. Li, Y. Xu, and M. Q.-H. Meng, “Automatic Recognition of Abdominal Organs in Ultrasound Images based on Deep Neural Networks and K- Nearest-Neighbor Classification,” in 2021 IEEE International Conference on Robotics and Biomimetics (ROBIO), Oct. 2021.

[0210]

[0028] S. Gerlach et al., “Robotic ultrasound-guided SBRT of the prostate: feasibility with respect to plan quality,” Int J Comput Assist Radiol Surg, vol. 12, no. 1, pp.149-159, Jan. 2017.

[0211]

[0029] S. Ipsen et al., “A Visual Probe Positioning Tool for 4D Ultrasound- Guided Radiotherapy,” 40th Annual International Conference of the IEEE Engineering in Medicine and Biology Society (EMBC), 2018.

[0212]

[0030] D. Sakabe et al., “Image quality characteristics for virtual monoenergetic images using dual-layer spectral detector CT: Comparison with conventional tube-voltage images,” Physica Medica, vol. 49, pp. 5-10, May 2018.

[0213]

[0031] A. Chandler, W. Wei, D. H. Herron, E. F. Anderson, V. E. Johnson, and C. S. Ng, “Semiautomated Motion Correction of Tumors in Lung CT-perfusion Studies,” Acad Radiol, vol. 18, no. 3, pp. 286-293, Mar. 2011.

[0214]

[0032] J.-M. Peyrat, H. Delingette, M. Sermesant, C. Xu, S. Member, and N. Ayache, “Registration of 4D Cardiac CT Sequences Under Trajectory Constraints With Multichannel Diffeomorphic Demons,” IEEE Trans Med Imaging, 2009.

[0215]

[0033] G. Meschini et al., “A clustering approach to 4D MRI retrospective sorting for the investigation of different surrogates,” Physica Medica, vol. 58, pp. 107-113, Feb. 2019.

[0216]

[0034] G. S. V. N. Chilla, C. H. Tan, and C. L. Poh, “Deformable registrationbased super-resolution for isotropic reconstruction of 4-D MRI volumes,” IEEE J Biomed Health Inform, vol. 21, no. 6, pp. 1617-1624, Nov. 2017.UMN 2025-187-02Quarles 920171.00690

[0217]

[0035] T. Kustner et al., “LAPNet: Non-Rigid Registration Derived in k-Space for Magnetic Resonance Imaging,” IEEE Trans Med Imaging, vol. 40, no. 12, pp. 3686–3697, Dec. 2021.

[0218]

[0030] J. S. Heiselman, W. R. Jarnagin, and M. I. Miga, “Intraoperative Correction of Liver Deformation Using Sparse Surface and Vascular Features via Linearized Iterative Boundary Reconstruction,” IEEE Trans Med Imaging, vol. 39, no. 6, pp. 2223–2234, Jun. 2020.

[0219]

[0037] Y. Zhou et al., “Correlation-Weighted Sparse Representation for Robust Liver DCE-MRI Decomposition Registration,” IEEE Trans Med Imaging, vol. 38, no. 10, pp.2352-2363, Oct. 2019.

[0220]

[0038] N. Paragios, “A level set approach for shape-driven segmentation and tracking of the left ventricle,” IEEE Trans Med Imaging, vol. 22, no. 6, pp. 773-776, Jun. 2003.

[0221]

[0039] M. Buehler, J. M. Slagowski, C. A. Mistretta, C. M. Strother, and M. A. Speidel, “4D DS A reconstruction using tomosynthesis projections,” in Medical Imaging 2017.

[0222]

[0040] R. Li et al., “3D tumor localization through real-time volumetric x-ray imaging for lung cancer radiotherapy,” Med Phys, vol. 38, no. 5, pp. 2783-2794, 2011.

[0223]

[0041] D. Wulff, I. Kuhlemann, F. Ernst, A. Schweikard, and S. Ipsen, “Robust motion tracking of deformable targets in the liver using binary feature libraries in 4D ultrasound,” Current Directions in Biomedical Engineering, vol. 5, no. 1, pp. 601-604, Sep.2019.

[0224]

[0042] V. De Luca et al., “Evaluation of 2D and 3D ultrasound tracking algorithms and impact on ultrasound-guided liver radiotherapy margins,” Med Phys, vol. 45, no. 11, pp. 4986-5003, Nov. 2018.

[0225]

[0043] M. De Craene et al., “3D strain assessment in ultrasound (straus): A synthetic comparison of five tracking methodologies,” Transactions on Medical Imaging, vol.32, no. 9, 2013.

[0226]

[0044] B. Heyde et al., “Elastic image registration versus speckle tracking for 2-d myocardial motion estimation: A direct comparison in vivo,” IEEE Trans Med Imaging, vol. 32, no. 2, pp. 449-459, 2013.UMN 2025-187-02Quarles 920171.00690

[0227]

[0045] J. R. McClelland, D. J. Hawkes, T. Schaeffter, and A. P. King, “Respiratory motion models: A review,” Medical Image Analysis, vol. 17, no. 1. pp. 19-42, Jan. 2013.

[0228]

[0046] B. Heiling et al., “Evaluating Diagnostic Ultrasound of the Vagus Nerve as a Surrogate Marker for Autonomic Neuropathy in Diabetic Patients,” Medicina (B Aires), vol. 59, no. 3, p. 525, Mar. 2023, doi: 10.3390 / medic- ina59030525.

[0229]

[0047] P. S. Olofsson and K. J. Tracey, “Bioelectronic medicine: technology targeting molecular mechanisms for therapy,” J Intern Med, vol. 282, no. 1, pp. 3-4, Jul. 2017, doi: 10.1111 / joim.12624.

[0230]

[0048] K. Birmingham et al., “Bioelectronic medicines: A research roadmap,” Nature Reviews Drug Discovery, vol. 13, no. 6. Nature Publishing Group, pp. 399-400, 2014. doi: 10.1038 / nrd4351.

[0231]

[0049] C. J. Wright, S. R. Haqshenas, J. Rothwell, andN. Saffari, “Unmyelinated Peripheral Nerves Can Be Stimulated in Vitro Using Pulsed Ultrasound,” Ultrasound Med Biol, vol. 43, no. 10, pp. 2269-2283, Oct. 2017, doi:10.1016 / j.ultrasmedbio.2017.05.008.

[0232]

[0050] V. Colucci, G. Strichartz, F. Jolesz, N. Vykhodtseva, and K. Hynynen, “Focused Ultrasound Effects on Nerve Action Potential in vitro,” Ultrasound Med Biol, vol.35, no. 10, pp. 1737-1747, Oct. 2009, doi: 10.1016 / j.ultrasmedbio.2009.05.002

[0233]

[0051] M. E. Downs, S. A. Lee, G. Yang, S. Kim, Q. Wang, and E. E. Konofagou, “Non-invasive peripheral nerve stimulation via focused ultrasound in vivo,” Phys Med Biol, vol. 63, no. 3, Feb. 2018, doi: 10.1088 / 1361-6560 / aa9fc2.

[0234]

[0052] S. K. Zhou et al., “A Review of Deep Learning in Medical Imaging: Imaging Traits, Technology Trends, Case Studies with Progress Highlights, and Future Promises,” Proceedings of the IEEE, vol. 109, no. 5, pp. 820-838, May 2021.

[0235]

[0053] Y. Fu, Y. Lei, T. Wang, W. J. Curran, T. Liu, and X. Yang, “Deep learning in medical image registration: A review,” Physics in Medicine and Biology, vol. 65, no. 20. IOP Publishing Ltd, Oct. 21, 2020.

[0236]

[0054] F. Altaf, S. M. S. Islam, N. Akhtar, and N. K. Janjua, “Going deep in medical image analysis: Concepts, methods, challenges, and future directions,” IEEE Access, vol. 7. Institute of Electrical and Electronics Engineers Inc., pp. 99540-99572, 2019.UMN 2025-187-02Quarles 920171.00690

[0237]

[0055] D. J. Lin, P. M. Johnson, F. Knoll, and Y. W. Lui, “Artificial Intelligence for MR Image Reconstruction: An Overview for Clinicians,” Journal of Magnetic Resonance Imaging, vol. 53, no. 4. John Wiley and Sons Inc, pp. 1015-1028, Apr. 01, 2021.

[0238]

[0056] K. A. Philbrick et al., “RIL-Contour: a Medical Imaging Dataset Annotation Tool for and with Deep Learning,” J Digit Imaging, vol. 32, no. 4, pp. 571-581, Aug. 2019.

[0239]

[0057] A. X. Chang et al., “ShapeNet: An Information-Rich 3D Model Repository,” In arXiv Preprint, Dec. 2015.

[0240]

[0058] S. T. Koch Berlin skoch et al., “ABC: A Big CAD Model Dataset For Geometric Deep Learning,” in Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, Proceedings of the IEEE / CVF conference on computer vision and pattern recognition, 2019.

[0241]

[0059] J. Yang, U. Wickramasinghe, B. Ni, and P. Fua, “ImplicitAtlas: Learning Deformable Shape Templates in Medical Imaging,” in Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition., 2022.

[0242]

[0060] Q. Quan, Q. Yao, J. Li, and S. K. Zhou, “Which images to label for fewshot medical landmark detection?” Proceedings of the IEEE / CVF Conference on Computer Vision and Pattern Recognition, 2022.

[0243]

[0061] X. Liang, S. Lin, F. Liu, D. Schreiber, and M. Yip, “ORRN: An ODE-based Recursive Registration Network for Deformable Respiratory Motion Estimation with Lung 4DCT Images,” IEEE Trans Biomed Eng, May 2023.

[0244]

[0062] V. De Luca, M. Tschannen, G. Sz'ekely, and C. Tanner, “LNCS 8149 - A Learning-Based Approach for Fast and Robust Vessel Tracking in Long Ultrasound Sequences,” in Medical Image Computing and Computer- Assisted Intervention-MICCAI 2013: 16th International Conference, Nagoya, Japan, September 22-26, 2013, Proceedings, Part I 16. Springer Berlin Heidelberg, 2013., 2013.

[0245]

[0063] I. M. Miki'c, S. Krucinski, and J. D. Thomas, “Segmentation and Tracking in Echocardiographic Sequences: Active Contours Guided by Optical Flow Estimates,” IEEE Trans Med Imaging, vol. 17, no. 2, 1998.

[0246]

[0064] H. Rivaz and D. L. Collins, “Near Real-Time Robust Non-rigid Registration of Volumetric Ultrasound Images for Neurosurgery,” Ultrasound Med Biol, vol.41, no. 2, pp. 574-587, 2015.UMN 2025-187-02Quarles 920171.00690

[0247]

[0065] M. G. A. B. Xiaohui Zhang, “Real-Time Organ Tracking in Ultrasound Imaging Using Active Contours and Conditional Density Propagation,” Medical Imaging and Augmented Reality, vol. 6325, 2010.

[0248]

[0066] J. Shi and C. Tomasi, “Good features to track,” in Proceedings of the IEEE Computer Society Conference on Computer Vision and Pattern Recognition, Publ by IEEE, 1994, pp. 593-600.

[0249]

[0067] C. Tomasi and T. Kanade, “Shape and Motion from Image Streams: a Factorization Method — Part 3 Detection and Tracking of Point Features,” Proceedings of the National Academy of Sciences 90.21, 1991.

[0250]

[0068] S. Oron, A. Bar-Hillel, and S. Avidan, “LNCS 8693 - Extended Lucas- Kanade Tracking,” in Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014.

[0251]

[0069] Y. et al Ahmine, “Adaptive Lucas-Kanade Tracking,” Image and Vision Computing, 88, 1-8, 2019.

[0252]

[0070] H. Bristow and S. Lucey, “In Defense of Gradient-Based Alignment on Densely Sampled Sparse Features,” Springer, Cham. doi:10.1007 / 978-3- 319-23048-1-7, 2016.

[0253]

[0071] C.-H. Lin, R. Zhu, and S. Lucey, “The Conditional Lucas Kanade Algorithm,” in In Computer Vision-ECCV 2016: 14th European Conference, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part V 14 (pp. 793-808). Springer International Publishing., Mar. 2016.

[0254]

[0072] C.-H. Chang, C.-N. Chou, and E. Y. Chang, “CLKN: Cascaded Lucas- Kanade Networks fo

[0105] age Alignment,” in Proceedings of the IEEE conference on computer vision and pattern recognition, 2017.

[0255]

[0073] Z. Kalal, K. Mikolajczyk, and J. Matas, “Forward-backward error: Automatic detection of tracking failures,” in Proceedings - International Conference on Pattern Recognition, 2010, pp. 2756-2759.

[0256]

[0074] L. Petrusca et al., “Hybrid Ultrasound / Magnetic Resonance Simultaneous Acquisition and Image Fusion for Motion Monitoring in the Upper Abdomen,” Investigative radiology 48.5 (2013): 333-340, 2013.

[0257]

[0075] De Luca, V., et al. “The 2014 liver ultrasound tracking benchmark.” Physics in Medicine and Biology 60.14, 2015: 5571.UMN 2025-187-02Quarles 920171.00690

[0258]

[0076] De Luca, Valeria, et al. “Evaluation of 2D and 3D ultrasound tracking algorithms and impact on ultrasound-guided liver radiotherapy margins” Medical physics 45.11, 2018: 4986-5003.

[0259]

[0077] Shepard, Andrew J., et al. “A block matching based approach with multiple simultaneous templates for the real-time 2D ultrasound tracking of liver vessels”, Medical physics 44.11, 2017: 5889-5900.

[0260]

[0078] Hallack A, Papiez BW, Cifor A, Gooding MJ, Schnabel JA. “Robust Liver Ultrasound Tracking using Dense Distinctive Image Features”, MICCAI 2015 Challenge on Liver Ultrasound Tracking. Munich, Germany; 2015.

[0261]

[0079] Gomariz, Alvaro, et al. “Siamese networks with location prior for landmark tracking in liver ultrasound sequences.” 2019 IEEE 16th International Symposium on Biomedical Imaging (ISBI 2019). IEEE, 2019.

[0262]

[0080] Guo H, et al. “Ultrasound does not activate but can inhibit in vivo mammalian nerves across a wide range of parameters.” Sci Rep. 2022 Feb 9; 12(1):2182.

[0263]

[0081] Rodríguez-Meana B, et al. “Assessment of focused ultrasound stimulation to induce peripheral nerve activity and potential damage in vivo.” Front Neurol. 2024 Feb 28;15:1346412.

[0264] A number of references to patent and non-patent literature are made throughout the specification, each of which is incorporated by reference in its entirety.

[0265] Thus, while the invention has been described above in connection with particular embodiments and examples, the invention is not necessarily so limited, and that numerous other embodiments, examples, uses, modifications and departures from the embodiments, examples and uses are intended to be encompassed by the claims attached hereto.

Claims

UMN 2025-187-02Quarles 920171.00690 CLAIMSWhat is claimed is:

1. A method of tracking motion of an organ, comprising:receiving an image from a time series of images of the organ collected by an imaging device;selecting a region of interest (ROI) of the image and identifying features within the ROI;tracking the identified features from the image to a subsequent image collected later in the time series;determining a confidence level of the tracking and assessing whether the confidence level is acceptable,wherein, if the confidence level is not acceptable, a trained network is used to predict a location of the ROI in a set of subsequent images collected over a time range, and resuming tracking and determining the confidence level of the tracking after the time range; andoutputting a trajectory of the organ over the time series of images.

2. The method of claim 1, wherein the ROI contains texture information from which the features can be identified based on a gradient distribution of a structure tensor of the ROI.

3. The method of claim 1 or 2, wherein the trained network used to predict the location of the ROI in a set of images over a time range comprises a long short-term memory (LSTM) algorithm, configured to have an architecture comprising layers and hidden units.

4. The method of any one of claims 1-3, wherein the time range over which the location of the ROI is predicted is equal to or greater than a duration of one breath cycle.

5. The method of claim 1, wherein the method further comprises adjusting the location of the ROI to a location with a greater number of features for tracking.UMN 2025-187-02Quarles 920171.00690 6. The method of claim 5, wherein the method further comprises delivering a therapy to the organ with a therapy device comprising at least one of a motion sensor, vibration sensor, respirometer, heart rate sensor, the imaging device, or another imaging device.

7. The method of any one of claims 1-6, wherein the organ comprises an abdominal organ, and the abdominal organ is selected from a group comprising liver, spleen, stomach, small intestine, large intestine, gallbladder, pancreas, adrenal gland, uterus, fallopian tubes, ovaries, kidneys, ureters, and bladder.

8. The method of any one of claims 1-7, wherein multiple ROIs are tracked.

9. The method of any one of claims 1-8, wherein multiple organs are tracked.

10. The method of any one of claims 1-9, wherein assessing whether the confidence level is acceptable comprises determining a confidence score and assessing whether the confidence score is greater than a threshold.

11. The method of any one of claims 1-9, wherein assessing whether the confidence level is acceptable comprises receiving user input indicating whether the confidence level is acceptable.

12. The method of any one of claims 1-11, wherein the imaging device comprises at least one of an ultrasound device, a magnetic resonance imaging device, or a fluoroscopy device.

13. A system for tracking motion of an organ, comprising:an imaging device; anda processor in communication with a memory, the memory having stored thereon a set of instructions which, when executed by the processor, cause the processor to:receive an image from a time series of images of the organ collected by the imaging device,select a region of interest (ROI) and identify features within the ROI, track the identified features from the image to a subsequent image collected later in the time series,UMN 2025-187-02Quarles 920171.00690 determine a confidence level of the tracking and assess whether the confidence level is acceptable,wherein, if the confidence level is not acceptable, a trained network is used to predict a location of the ROI in a set of subsequent images collected over a time range, and resuming tracking and determining the confidence level of the tracking after the time range, andoutput a trajectory of the organ over the time series of images.

14. The system of claim 13, wherein the ROI contains texture information from which the features can be identified based on a gradient distribution of a structure tensor of the ROI.

15. The system of claim 13 or 14, wherein the trained network used to predict the location of the ROI in a set of images over a time range comprises a long short-term memory (LSTM) algorithm, configured to have an architecture comprising layers and hidden units.

16. The system any one of claims 13-15, wherein the time range over which the location of the ROI is forecasted is equal to or greater than a duration of one breath cycle.

17. The system of claim 13, wherein the set of instructions further cause the processor to adjust the location of the ROI to a location with a greater number of features for tracking.

18. The system of claim 17, wherein the system is further configured to deliver a therapy to the organ with a therapy device comprising at least one of a motion sensor, vibration sensor, respirometer, heart rate sensor, the imaging device, or another imaging device19. The system of any one of claims 13-18, wherein the organ is an abdominal organ, and the abdominal organ is selected from a group comprising liver, spleen, stomach, small intestine, large intestine, gallbladder, pancreas, adrenal gland, uterus, fallopian tubes, ovaries, kidneys, ureters, and bladder.

20. The system of any one of claims 13-19, wherein multiple ROIs are tracked.

21. The system of any one of claims 13-20, wherein multiple organs are tracked.UMN 2025-187-02Quarles 920171.0069022. The system of any one of claims 13-21, wherein assessing whether the confidence level is acceptable comprises determining a confidence score and assessing whether the confidence score is greater than a threshold.

23. The system of any one of claims 13-21, wherein assessing whether the confidence level is acceptable comprises receiving user input indicating whether the confidence level is acceptable.

24. The system of any one of claims 13-23, wherein the imaging device comprises at least one of an ultrasound device, a magnetic resonance imaging device, or a fluoroscopy device.

Images