Robotized us imaging system

Abstract

Systems and methods are disclosed for autonomously recording one or more ultrasound images during an ultrasound procedure. The method comprises controlling a robotic system to move an ultrasound probe relative to a patient's body and receiving a plurality of ultrasound frames from the ultrasound probe during the movement of the ultrasound probe. Each of the plurality of ultrasound frames is associated with a respective ultrasound probe pose. The method further comprises determining a quality score for each of the plurality of ultrasound frames and identifying a recording pose. The method may further comprise controlling a robotic system to move the ultrasound probe relative to a patient's body, based on an image-based search algorithm using as input at least one of: the determined quality scores and the plurality of ultrasound frames, and, optionally, the respective ultrasound probe poses. The method further comprises identifying a recording pose using the determined quality scores, the recording pose being a first respective ultrasound probe pose associated with a first ultrasound frame of the plurality of ultrasound frames. The method further comprises controlling the robotic system to move the ultrasound probe to the recording pose and recording the one or more ultrasound images while the ultrasound probe is positioned at the identified recording pose.

Description

[0001]

[0002] Robotized US imaging system

[0003] Technical field

[0004] This disclosure relates to a system and method for ultrasound imaging, and in particular, though not exclusively, to a system and method for autonomous ultrasound imaging using a robotic ultrasound system. This disclosure further relates to a computer- implemented method for autonomous ultrasound imaging and to a computer program and computer-readable storage medium for performing such method.

[0005] Background

[0006] Cardiovascular diseases remain the leading cause of death globally, driving the demand for precise and timely diagnostic imaging, particularly in the form of echocardiography. Echocardiography, or ultrasound imaging of the heart, is a vital tool for diagnosing heart conditions such as valve disorders, heart failure, and other structural abnormalities. Traditional echocardiographic procedures involve the use of handheld ultrasound probes, which are manually positioned by a skilled sonographer to capture various views of the heart. These views typically include the parasternal, apical, subcostal, and suprasternal windows, each of which requires careful probe manipulation and orientation to ensure clear visualization of specific anatomical features.

[0007] The manual nature of these procedures presents several challenges. First, sonographers must reposition the patient frequently to avoid interference from surrounding tissues like the ribs and lungs, which can obscure critical cardiac structures in certain views. This requires expert knowledge of both cardiac anatomy and optimal probe positioning to ensure accurate imaging. Second, the process is labour-intensive and time-consuming, as up to 37 different views may be required to fully assess the heart, depending on the complexity of the patient’s condition. Given the global shortage of skilled sonographers, patients often face long waiting times for echocardiographic exams, which delays diagnosis and treatment. Additionally, human error in probe positioning or image interpretation can lead to incomplete or suboptimal imaging, potentially impacting the accuracy of diagnoses.

[0008] Efforts to alleviate these issues have led to the development of robotic ultrasound systems, such as the one described in US2024 / 0115235A1. This system introduces a robotized ultrasound probe that is controlled autonomously using machine learning algorithms. The robotic system is designed to move the probe over the patient’s body to capture echocardiographic images at predefined anatomical views. It uses direction information based on images captured at various probe positions and orientations to guide the robotic arm to an optimal position for capturing each view. By automating this process, the system reduces the need for human intervention and ensures that the probe is positioned correctly in relation to the patient’s anatomy. However, while the robotic system described in US2024 / 0115235A1 represents an important step towards automating echocardiography, certain limitations remain. Although the system employs a trained machine-learning model to automate B-mode image acquisition by determining an optimal probe position and orientation for each target view, challenges remain in consistently obtaining high-quality images. In some cases, the system may produce images of varying quality or fail to locate an optimal probe position, especially in anatomically complex or difficult scanning scenarios.

[0009] Non-B-mode ultrasound modalities, such as Colour Doppler, Pulsed-Wave Doppler, M- mode, and elastography, provide critical diagnostic information by measuring dynamic processes like blood flow, tissue motion, and elasticity. Traditionally, the optimization of these modalities requires manual adjustments by a sonographer, who must fine-tune parameters such as Doppler gain, sample volume, sample volume placement and pulse repetition frequency (PRF) to ensure high-quality images.

[0010] In addition to automating B-mode imaging, which provides 2D grayscale images or colorized B-mode images of anatomical structures, US2022 / 296219A1 further proposed processing different non-B-mode ultrasound images through dedicated mode-related modules to enhance imaging guidance during medical procedures. US2022 / 296219A1 identifies the need to separately optimise non-B-mode imaging modalities (such as Colour Doppler, M-mode, spectral Doppler, elastography, etc.) but does not but does not teach how that may be done. Similarly, US2024 / 0115235A1 does not provide a solution to the autonomous acquisition and optimization of non-B-mode ultrasound images.

[0011] Therefore, from the above, it follows that there is a need for further advancements in robotic ultrasound imaging systems that can address these limitations, capturing high-quality B-mode and non-B-mode echocardiographic images autonomously or with minimal human intervention.

[0012] It is an aim of embodiments in this disclosure to provide a system and method for autonomously recording ultrasound images that avoids, or at least reduces the drawbacks of the prior art.

[0013] In an aspect, embodiments in this disclosure relate to a computer-implemented method for autonomously recording one or more ultrasound images during an ultrasound procedure, in particular of a (living) heart, more in particular a human heart. The method comprises receiving a plurality of ultrasound frames from an ultrasound probe. Each of the plurality of ultrasound frames is associated with a respective ultrasound probe pose, and may further be associated with a respective cardiac and / or respiratory phase. The probe pose may be defined in a probe space, generally a multidimensional space. The method further comprises determining a quality score for each of the plurality of ultrasound frames. The quality score may indicate a diagnostic suitability of the ultrasound frame, and / or may be representative of (e.g., indicate, comprise, or be based on) a likelihood that the ultrasound frame corresponds to a target anatomical view. The method further comprises controlling a robotic system to move the ultrasound probe relative to a patient’s body, based on a search algorithm. The search algorithm may be an image-based search algorithm using as input at least one of the determined quality scores and the plurality of ultrasound frames. Optionally, the search algorithm may (also) use the respective ultrasound probe poses as input. The method further comprises identifying a recording pose using the determined quality scores, the recording pose being a first respective ultrasound probe pose associated with a first ultrasound frame of the plurality of ultrasound frames. The method further comprises controlling the robotic system to move the ultrasound probe to the recording pose and recording the one or more ultrasound images while the ultrasound probe is positioned at the identified recording pose. The one or more ultrasound images may comprise one or more ultrasound frames associated with one or more further cardiac and / or respiratory phases, e.g., a cardiac and / or respiratory phase different from the cardiac and / or respiratory phase associated with the ultrasound frame associated with the recording pose.

[0014] Controlling the robotic system may comprise determining one or more control signals and sending the one or more control signals to the robotic system, e.g., via communication interface. The robotic system may be configured to receive the one or more control signals and to act in response to the received one or more control signals, e.g., to move according to a path, in a direction, or to a predetermined point encoded by the one or more signals, to change the orientation of the probe, to capture an ultrasound signal, and so on.

[0015] The transition from controlling the robotic system to move the ultrasound probe based on a search algorithm to identifying the recording pose may be based on a stopping criterion, e.g., a time limit, a step limit, meeting a minimum quality score, or not increasing a maximum quality score over a predetermined amount of time or frames.

[0016] By associating each ultrasound frame with a pose, and determining a quality score for each ultrasound frame, effectively, a quality score for each recording pose may be determined. This allows optimisation of the recording pose using various optimisation techniques. It is noted that even when a full ultrasound image may contain several frames (and hence several seconds to acquire), a quality score may be determined for each separate frame, allowing optimising the recording pose in an acceptable amount of time.

[0017] When an ultrasound image of the heart is obtained, different frames are associated with different cardiac phases and with different respiratory phases. Thus, each individual frame is not representative of the entire cardiac cycle, and hence, care should be taken that when selecting a recording pose, that this recording pose is clinically suitable for at least the cardiac phase to be imaged, which often includes an entire cardiac cycle. Information regarding the cardiac and / or respiratory phase may be implicitly or explicitly included in the quality score. Additionally or alternatively, information regarding the cardiac and / or respiratory phase (e.g., obtained from either the ultrasound frames or from an external source such as a dedicated sensor signal) may be used to interpret the quality score, using, e.g., an implicit or explicit weigh to determine a comprehensive quality score based on an initial quality score (based on the ultrasound frame, but not explicitly taking cardiac and / or respiratory phase into account) and a signal representing the cardiac and / or respiratory phase, or in some cases a different physiological signal. Compared to systems that only use image data, an optimal recording pose may be determined more reliably and / or more efficiently. An advantage of this approach is that the ultrasound probe pose associated with an ultrasound frame can be used to guide the ultrasound probe movement, ensuring the robotic system adjusts the ultrasound probe relative to an optimal pose, determined during the ultrasound probe movement. This method helps prevent revisiting the same location, thereby improving efficiency. Additionally, it enables the correlation of ultrasound images captured at the same location or pose over time, providing valuable context for comparison. If a particular ultrasound image is later determined to be an optimal one, the robotic system can return to its exact location or pose. Furthermore, once a ultrasound image is recorded at the recording pose, this pose can serve as a reference point to infer the starting position for searching other anatomical views, facilitating a more streamlined and systematic search process.

[0018] In an embodiment, the image-based search algorithm is configured to at least explore probe poses in the probe space outside of a predetermined scanning path. Thus, the imagebased search algorithm may be iterative and / or have multiple stages, and may be configured to explore the probe space, including probe poses not visited in a previous iteration and / or stage.

[0019] In an embodiment, each of the one or more ultrasound images comprises one or more frames. For example, in cardiac ultrasound, a length (in frames) of an ultrasound image or clip is typically dependent on a heartbeat, and may comprise, e.g., four heartbeats. In other applications, ultrasound images may contain a number of frames based on other criteria, such as breathing motion, foetal heartbeats, and so on. Yet other applications may use images comprising only a single frame.

[0020] In an embodiment, the quality score is further determined using additional sensor data. Such additional sensor data may comprise physiological signals (i.e. , signals representative of a physiological state of the subject being scanned); for example, the additional sensor data may comprise ECG data, which is indicative of the cardiac and / or respiratory phase. As mentioned above, such additional sensor data may be used to determine the quality score (in addition to the image data), or to interpret (e.g., weigh) the image-based quality score.

[0021] In an embodiment, the ultrasound probe pose defines a location and / or an orientation. The location can be, e.g., a two- or three-dimensional location, and may be defined by spatial coordinates. The orientation may be defined by angular coordinates, and may comprise, e.g., a pitch angle, a yaw angle, and a roll angle. In some embodiments, location and orientation may be optimised independently. For example, first the location may be optimised, and then the orientation.

[0022] In an embodiment, the quality score is based on an output from a machine learning algorithm. The machine learning algorithm may be trained to either classify input ultrasound frames into a plurality of predefined anatomical views, or to regress a likelihood score indicating the degree to which an ultrasound frame corresponds to a target anatomical view. The output of the machine learning algorithm may represent either a classification confidence score indicating the likelihood that an input ultrasound frame corresponds to one of the plurality of predefined anatomical views or a regressed likelihood score. For example, the machine learning algorithm can be a classifier algorithm, trained to classify ultrasound frames into a plurality of categories corresponding to different (desired) anatomical views, and optionally a category for frames not corresponding to any of the anatomical views. The quality score may then be based on the score assigned to the category corresponding to the anatomical view under consideration. Alternatively, the machine learning algorithm can be a regression algorithm, trained to predict a continuous likelihood score for each ultrasound frame based on features extracted from the ultrasound image. This score may represent the likelihood of the ultrasound frame corresponding to a target anatomical view or another score indicating an overall quality of the ultrasound frame. The quality score may account for a plurality of factors, such as image noise, contrast, sharpness, the visibility of anatomical features, and / or alignment with the target anatomical view.

[0023] In an embodiment, the image-based search algorithm may be selected from a collection of search algorithms. The collection of search algorithms can comprise one or more of: a direct search algorithm, an adaptive search algorithm, or a feature-based search algorithm.

[0024] The direct search algorithm may use the one or more ultrasound frames of the plurality of ultrasound frames as input to a navigation machine-learning model to predict a target pose corresponding to the target anatomical view relative to the one or more respective ultrasound poses associated with the one or more ultrasound frames. For example, the direct search algorithm may utilize a navigation machine-learning model, which analyses the captured ultrasound frames to estimate lateral motion (e.g., in the x and y direction) needed to locate a target body part (e.g., the heart, lungs, uterus, other organs). This search algorithm may be designed to perform coarse adjustments, enabling the robotic system to efficiently approximate the location of a target body part, even if the received ultrasound frames do not capture (detailed) anatomical features of the target body part.

[0025] The adaptive search algorithm may define one or more second movement patterns for moving the ultrasound probe relative to the patient’s body. The one or more second movement patterns may comprise movements along multiple degrees of freedom, e.g., simultaneously or subsequently. The one or more second movement patterns may be dynamically adjusted based on the quality scores determined for the plurality of ultrasound frames obtained during execution of the adaptive search algorithm. Examples of an adaptive search algorithm include so-called hill climber algorithms, gradient-ascent algorithms, and the like.

[0026] In an embodiment, the feature-based search algorithm defines one or more third movement patterns for moving the ultrasound probe relative to the patient’s body. The one or more third movement patterns may be determined based on one or more localized anatomical features in the plurality of ultrasound frames. The one or more localized anatomical features may be determined using a feature-detection algorithm.

[0027] The one or more localized anatomical features may comprise ‘wanted’ features, which should be present in the ultrasound image (e.g., heart chambers, or valves), and / or ‘unwanted’ features which should be avoided (e.g., ribs, or air pockets). By localising the wanted and / or unwanted anatomical features, recoding poses may be determined that have a high likelihood to provide images including the wanted features and / or that have a low likelihood to provide images including the unwanted features.

[0028] In an embodiment, the feature-detection algorithm comprises a machine-learning model such as an object-detection algorithm or an instance-segmentation algorithm. The machine-learning model may include one or more deep neural networks. The machinelearning model may be trained using an input dataset comprising ultrasound images and corresponding target locations for the one or more localized anatomical features in each ultrasound image. The feature-detection algorithm may output a predicted location and / or a confidence score for each of the one or more localized anatomical features.

[0029] In an embodiment, controlling the robotic system to move the ultrasound probe relative to the patient’s body comprises moving the ultrasound probe according to a first search algorithm from the collection of search algorithms until a first stopping criterion is met, and moving the ultrasound probe according to a second search algorithm from the collection of search algorithms until a second stopping criterion is met.

[0030] The first search algorithm may either be selected from the collection of search algorithms or the first search algorithm may be a deterministic search algorithm. Such a deterministic search algorithm may define one or more first movement patterns for moving the ultrasound probe relative to the patient’s body. The one or more first movement patterns may comprise e.g., a spiral pattern and / or a lawnmower pattern.

[0031] Deterministic search algorithms are relatively simple to implement, and are particularly useful for quickly obtaining a rough landscape of recording pose quality. In some embodiments, the deterministic search algorithm may only adjust the location of the probe, keeping the orientation constant.

[0032] The first and second stopping criteria may be similar or dissimilar to each other. For example, the first stopping criterion may be a first quality threshold, while the second stopping criterion is a time limit or iteration limit. The second stopping criterion may be related to or depend on the first stopping criterion. For example, the second stopping criterion may be a second quality threshold higher than the first quality threshold. More than two search algorithms may be combined. The decision whether or not to use a second or further search algorithm may depend on, e.g., the quality scores of the received ultrasound frames, and / or on the time already spent searching.

[0033] Thus, various search algorithms may be combined. For example, first a relatively fast deterministic search algorithm may be used to determine an appropriate starting point, followed by a slower, but more accurate adaptive search algorithm.

[0034] In a further aspect, embodiments in this disclosure relate to a method for autonomously recording one or more non-B-mode ultrasound image during an ultrasound procedure. The method comprises receiving at least one B-mode ultrasound image, generated by an ultrasound probe, the at least one B-mode ultrasound image corresponding to a target anatomical view. A non-B-mode imaging modality is received or may be determined. The method further comprises determining a plurality of beam paths and / or volumes for the received or determined non-B-mode imaging modality and determining a quality parameter for each determined beam path and / or volume. The method further comprises selecting a beam path and / or volume based on the quality parameter and based on the received non-B- mode imaging modality. The method further comprises recording one or more non-B-mode ultrasound images using the non-B-mode imaging modality and the selected beam path and / or volume.

[0035] The method may be a computer-implemented method.

[0036] Depending on the protocol, for a single view, both a B-mode image and one or more non-B-mode images may be acquired. These non-B-mode images may have specific requirements regarding, e.g., anatomical features that should be imaged, image depth, image width, contrast, et cetera.

[0037] In some embodiments, the desired non-B-mode images may be determined based on an image analysis of the B-mode image, based on the one or more anatomical features (which may be detected automatically, or received from, e.g., a user), or determined based on a stored list with desired views and image modalities.

[0038] In an embodiment, determining the plurality of beam paths and / or volumes is based on a statistical metric, e.g., a mean or median, computed using the locations of the one or more anatomical features within the at least one B-mode ultrasound image.

[0039] For example, the non-B-mode image may need a beam path intersecting with an anatomical feature, and the statistical metric may indicate a beam path that has a high likelihood of intersecting with the anatomical feature. In an embodiment, the at least one B- mode ultrasound image comprises a plurality of frames, and the statistical metric is determined based on the respective locations of the one or more anatomical features in the plurality of frames.

[0040] In an embodiment, determining a plurality of beam paths and / or volumes may be based on one or more of a feature-based method, a flow-based method, or a pattern-based method.

[0041] In the feature-based method, the plurality of beam paths and / or volumes may be generated using one or more locations corresponding to one or more anatomical features identified as outputs from a feature-detection algorithm applied to the B-mode ultrasound image.

[0042] In the flow-based method, the plurality of beam paths and / or volumes may be generated by sampling regions in the B-mode ultrasound image, the regions being identified based on flow velocity data. These flow velocity data may be derived from flow-related imaging modalities, such as colour flow mapping (CFM) of the received B-mode ultrasound image. For example, in a TTE exam, CFM enables the visualisation of blood flow direction and speed in the B-mode ultrasound image. By leveraging this data, the flow-based method can identify regions of interest within the B-mode ultrasound image that exhibit specific dynamic flow characteristics. The plurality of beam paths and / or volumes may then be sampled within these identified regions of interest.

[0043] In the pattern-based method, the plurality of beam paths and / or volumes are generated using a predetermined pattern around an initial estimate. In an embodiment, the initial estimate is derived from either the feature-detection algorithm applied to the B-mode ultrasound image, or from a different machine-learning model. In another embodiment, the initial estimate is received as user input.

[0044] In an embodiment, the one or more anatomical features within the B-mode ultrasound are determined using a feature-detection algorithm that comprises a machine-learning model, such as an object-detection algorithm or an instance-segmentation algorithm. The machine-learning model may include one or more deep neural networks. The machinelearning model may be trained using an input dataset comprising ultrasound images and corresponding target locations for the one or more anatomical features in each ultrasound image. The feature-detection algorithm may output, e.g., a predicted location for each of the one or more anatomical features, and / or a confidence score for each of the one or more anatomical features.

[0045] In an embodiment, the machine learning model used in the pattern-based method uses the feature-detection algorithm.

[0046] In another embodiment, the machine learning model used in the pattern-based method may include one or more architectures, such as convolutional neural networks (CNNs) and / or transformer-based models designed for image analysis, including Vision Transformerarchitectures.

[0047] The machine learning model may be trained using as input B-mode ultrasound images and one or more target beam paths and / or volumes corresponding to one or more non-B- mode ultrasound images. The machine learning model may be trained to optimize a loss function that computes an error between an output of the machine learning model, the output representing one or more predicted beam paths and / or volumes, and the corresponding target beam paths and / or volumes of the non-B-mode ultrasound images.

[0048] Such a machine learning model may or may not comprise the explicit determination of the one or more anatomical features (similar to the feature-detection algorithm). In particular, the machine learning model may determine the one or more beam paths and / or volumes directly based on the at least one B-mode image.

[0049] In an embodiment, the quality parameter is based on a measurement of a flow velocity and / or on a signal-to-noise ratio determined in the non-B-mode ultrasound image. This allows optimisation of the non-B-mode ultrasound image, or ensuring that the non-B-mode ultrasound image meets a predefined quality criterion.

[0050] In another embodiment, the quality parameter is determined using a non-B-mode quality-assessment model, which comprises a machine-learning model configured to classify or predict (via regression) the quality parameter of a non-B-mode ultrasound image. For example, the non-B-mode quality-assessment model may be trained on ultrasound sweep data to produce masks that segment velocity waveforms and estimate parameters such as the signal-to-noise ratio (SNR) and / or the maximum flow velocity based on the segmented masks.

[0051] In an embodiment, recording the one or more non-B-mode ultrasound images comprises a further non-B-mode image optimization process, wherein the non-B-mode image optimization comprises optimising one or more imaging parameters, such as sweep speed, baseline, sample volume, dynamic range, scale or gain. This can improve the non-B- mode ultrasound image quality, and / or reduce aliasing of the non-B-mode ultrasound images.

[0052] Optimising the one or more imaging parameters may comprise varying the one or more imaging parameters, determining a quality score for each image obtained with the varied one or more imaging parameters, and selecting the imaging parameter based on a maximisation of the quality score.

[0053] In an embodiment, the method comprises autonomously recording one or more ultrasound images during an ultrasound procedure as described herein, the one or more ultrasound images comprise a at least one first B-mode ultrasound image, and autonomously recording a non-B-mode ultrasound image as described herein, wherein the received at least one B-mode ultrasound image is the at least one first B-mode ultrasound image.

[0054] In a further aspect, this disclosure relates to robotic ultrasound imaging systems.

[0055] In an embodiment, a robotic ultrasound imaging systems comprises a robotic system configured to move an ultrasound probe relative to a patient’s body; and a computer system comprising a computer-readable storage medium storing computer-readable program code and a processor, e.g. a microprocessor, coupled to the computer-readable storage medium. Responsive to executing the computer-readable program code, the processor is configured to perform executable operations comprising: receiving a plurality of ultrasound frames from the ultrasound probe during the movement of the ultrasound probe, each of the plurality of ultrasound frames being associated with a respective ultrasound probe pose (e.g., in a probe space), and optionally each of the plurality of ultrasound frames being associated with a respective cardiac and / or respiratory phase; determining a quality score for each of the plurality of ultrasound frames, the quality score indicating a diagnostic suitability of the ultrasound frame and optionally the quality score being representative of a likelihood that the ultrasound frame corresponds to a target anatomical view; controlling a robotic system to move the ultrasound probe relative to a patient’s body, based on an image-based search algorithm using as input at least one of: the determined quality scores and the plurality of ultrasound frames, and, optionally, using as input the respective ultrasound probe poses; identifying a recording pose using the determined quality scores, the recording pose being a first respective ultrasound probe pose associated with a first ultrasound frame of the plurality of ultrasound frames; controlling the robotic system to move the ultrasound probe to the recording pose; and recording the one or more ultrasound images while the ultrasound probe is positioned at the identified recording pose, wherein the one or more ultrasound images may comprise one or more ultrasound frames associated with one or more further cardiac and / or respiratory phases.

[0056] In an embodiment, the one or more ultrasound images comprise a first B-mode ultrasound image. In such an embodiment, the processor may be configured to perform further executable operations comprising: receiving the first B-mode ultrasound image, generated by the ultrasound probe, the B-mode ultrasound image corresponding to a target anatomical view; receiving or determining a non-B-mode imaging modality; determining a plurality of beam paths and / or volumes based on the received or determined non-B-mode imaging modality; determining a quality parameter for each determined beam path and / or volume; selecting a beam path and / or a volume from the plurality of beam paths and / or volumes based on the determined quality parameter; and recording one or more non-B-mode ultrasound images using the non-B-mode imaging modality and the selected beam path and / or volume.

[0057] In an embodiment, the processor may be configured, responsive to executing the computer-readable program code, to perform executable operations comprising: receiving the first B-mode ultrasound image, generated by the ultrasound probe, the B-mode ultrasound image corresponding to a target anatomical view; receiving or determining a non- B-mode imaging modality; determining a plurality of beam paths and / or volumes for the non- B-mode imaging modality; determining a quality parameter for each determined beam path and / or volume; selecting a beam path and / or a volume from the plurality of beam paths and / or volumes based on the quality parameter; and recording one or more non-B-mode ultrasound images using the non-B-mode imaging modality and the selected beam path and / or volume.

[0058] In a further aspect, embodiments in this disclosure relate to a data processing system, e.g., a controller, comprising a computer-readable storage medium storing computer- readable program code and a processor, e.g. a microprocessor, coupled to the computer- readable storage medium. Responsive to executing the computer-readable program code, the processor is configured to perform the methods described herein. Such a data processing system may be included in a robotic ultrasound system as described above.

[0059] Such systems may perform the methods described herein.

[0060] As will be appreciated by one skilled in the art, aspects of the present invention may be embodied as a system, method or computer program product. Accordingly, aspects of the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment (including firmware, resident software, micro-code, etc.) or an embodiment combining software and hardware aspects that may all generally be referred to herein as a “circuit,” “module” or “system.” Functions described in this disclosure may be implemented as an algorithm executed by a microprocessor of a computer. Furthermore, aspects of the present invention may take the form of a computer program product embodied in one or more computer readable medium(s) having computer readable program code embodied, e.g., stored, thereon.

[0061] Any combination of one or more computer readable medium(s) may be utilized. The computer readable medium may be a computer readable signal medium or a computer readable storage medium. A computer readable storage medium may be, for example, but not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of the computer readable storage medium would include the following: an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or Flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this document, a computer readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device.

[0062] A computer readable signal medium may include a propagated data signal with computer readable program code embodied therein, for example, in baseband or as part of a carrier wave. Such a propagated signal may take any of a variety of forms, including, but not limited to, electro-magnetic, optical, or any suitable combination thereof. A computer readable signal medium may be any computer readable medium that is not a computer readable storage medium and that can communicate, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device.

[0063] Program code embodied on a computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber, cable, RF, etc., or any suitable combination of the foregoing. Computer program code for carrying out operations for aspects of the present invention may be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, Python, Smalltalk, C++ or the like and conventional procedural programming languages, such as the “C” programming language or similar programming languages. The program code may execute entirely on the user’s computer, partly on the user’s computer, as a stand-alone software package, partly on the user’s computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer may be connected to the user’s computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection may be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0064] Aspects of the present invention are described below with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor, in particular a microprocessor or central processing unit (CPU), of a general purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer, other programmable data processing apparatus, or other devices create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks.

[0065] These computer program instructions may also be stored in a computer readable medium that can direct a computer, other programmable data processing apparatus, or other devices to function in a particular manner, such that the instructions stored in the computer readable medium produce an article of manufacture including instructions which implement the function / act specified in the flowchart and / or block diagram block or blocks.

[0066] The computer program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other devices to cause a series of operational steps to be performed on the computer, other programmable apparatus or other devices to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide processes for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. Additionally, the Instructions may be executed by any type of processors, including but not limited to one or more digital signal processors (DSPs), general purpose microprocessors, application specific integrated circuits (ASICs), field programmable logic arrays (FPGAs), or other equivalent integrated or discrete logic circuitry.

[0067] The flowchart and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present invention. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the blocks may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams and / or flowchart illustrations, and combinations of blocks in the block diagrams and / or flowchart illustrations, can be implemented by special purpose hardware-based systems that perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

[0068] Brief description of the drawings

[0069] The embodiments will be further illustrated with reference to the attached schematic drawings, in which:

[0070] Fig. 1 illustrates a robotic ultrasound imaging system process according to an embodiment.

[0071] Fig. 2 depicts a flow-diagram of a computer-implemented method for autonomously recording one or more ultrasound images during an ultrasound procedure.

[0072] Fig. 3 represents a flowchart illustrating a method for controlling a robotic system to move an ultrasound probe relative to the patient’s body and to record an optimal ultrasound image, according to an embodiment.

[0073] Fig. 4 presents a detailed view of the search algorithms employed by the robotic system to search and record an optimal ultrasound image

[0074] Fig. 5 illustrates the method for autonomously recording one or more ultrasound images during an ultrasound procedure within a robotic ultrasound imaging system

[0075] Fig. 6 illustrates an example of the output of the feature-based search algorithm within a robotic ultrasound imaging system

[0076] Fig. 7 illustrates a flow-diagram describing the steps of a computer-implemented method for autonomously recording and optimizing non-B-mode ultrasound images during an ultrasound procedure. Fig. 8 illustrates an example of a non-B-mode ultrasound image recording performed autonomously by a robotic ultrasound imaging system during an echocardiography exam.

[0077] Fig. 9 illustrates the modules of a computer system 900 of a robotic ultrasound imaging system according to an embodiment.

[0078] Fig. 10 is a block diagram illustrating an exemplary data processing system that may be used for executing methods and software products described in this application.

[0079] Detailed description

[0080] In the figures, identical reference number indicate identical or at least similar elements. Fig. 1 illustrates a schematic of a robotic ultrasound imaging system 100 according to an embodiment. The robotic ultrasound imaging system 100 comprises a robotic system 124 and a computer system 102, for example a data processing system as described in more detail with reference to Fig. 10. The robotic system is configured to move an ultrasound probe 106 relative to a patient’s body 108 on which an ultrasound scan is performed. The patient may be a (living) human or animal. In some embodiments, the robotic system is configured to perform an ultrasound scan of a dynamic target, such as the heart. In the depicted embodiment, the robotic system includes a robot arm 104 for moving the ultrasound probe, and a robot controller for controlling the robot arm. In other embodiments, the robotic system may comprise other means for moving the ultrasound probe relative to the patient’s body; for example, a support structure supporting a body part to be imaged may be configured to move relative to a stationary ultrasound probe.

[0081] In this context, the term “patient” may refer to any person or animal undergoing an ultrasound procedure, and is used interchangeably with “subject”.

[0082] The computer system 102 is configured to autonomously control, directly or indirectly, the robotic system 124 to perform an ultrasound scan on the patient’s body 108. To this end, the computer system may be configured to perform method steps 112-122.

[0083] In some embodiments, the computer system 102 sends movement control commands 126 identifying a desired motion of the ultrasound probe 106 to a robot controller of the robotic system 124, causing the robot controller to move the ultrasound probe. The robot controller then translates the movement control commands into control signals for one or more actuators that cause the robotic system to move. For example, the movement control commands may identify an absolute or relative target pose of the ultrasound probe to the robotic system, or a direction of motion of the ultrasound probe, and possibly a speed of the ultrasound probe. In other embodiments, the computer system may send control signals to the actuators directly. In an embodiment, the robot controller may be embedded within the computer system, or alternatively, it may be a separate component within the robotic system.

[0084] Thus, the computer system 102 controls the robotic system to move the ultrasound probe 106 over the patient’s body 108. During the movement, the ultrasound captures a plurality of ultrasound frames at a plurality of respective ultrasound probe poses 110. The patient may be positioned on a support structure, e.g., a bed or examination table to achieve an appropriate pose for performing the ultrasound procedure, e.g., an echocardiography or other ultrasound procedure such as an abdominal ultrasound for examining abdominal organs, a vascular ultrasound procedure, a prenatal ultrasound procedure, etc. For example, during an echocardiography, the patient may be positioned in a prone, supine, left lateral decubitus, or other clinically relevant position to facilitate optimal image acquisition during an echocardiogram.

[0085] The computer system 102 may be configured to detect when the patient’s body part to be imaged is positioned outside the operational reach of the robotic system 124, such that the ultrasound probe is unable to establish contact. The detection may operate based on predefined spatial boundaries or range mappings that define the maximum reach limits of the robotic system, e.g., based on a length of the robotic arm 104. The computer system may be further configured to analyse whether any portion of the patient’s body extends beyond these boundaries. Additionally, the computer system may employ, e.g., image processing algorithms to identify unintended probe-to-patient contact, interruptions in ultrasound imaging coverage (e.g., due to insufficient gel), to ensure continuous and accurate imaging of the target area. In such cases, the computer system may prompt an operator or display instructions for solving the identified issue, e.g., by adjusting the patient’s position, applying additional gel, or ensuring that the ultrasound probe can adequately access a specific body part corresponding to a target anatomical view (e.g., a particular cardiac or abdominal structure) required for accurate ultrasound imaging.

[0086] The robotic system 124 may be arranged to position and move the ultrasonic probe 106 relative to the support system in alignment with the body part of the patient. In particular, the robotic system may be arranged to move (translate) the probe in three axial directions relative to the support system. The pose of the probe at a time instance may be defined based on suitable coordinate system, e.g., a cartesian coordinate system relative to the robotic system. The pose of the ultrasound probe, as controlled by the robotic arm, may include coordinates (x, y, z) corresponding to the translational movement of the ultrasound probe along three-dimensional axes. Further, the pose of the ultrasound probe may also include an angular component, defining its rotational orientation along a specified axis, e.g., relative to the support system. For example, the orientation of the ultrasound probe may be represented by Euler angles (a, p, y), corresponding to yaw, pitch, and roll, respectively, relative to the coordinate system axes. Thus, the movement of the ultrasound probe, i.e., the change in location and orientation at each time instance, may be described using the coordinates (x, y, z, a, p, y) where (x, y, z) denote the 3D location coordinates and (a, p, y) denote the orientation, or rotational orientation (yaw, pitch, and roll), collectively defining the probe’s pose comprising the probe’s location and the probe’s orientation. Other systems may have more, fewer, and / or different degrees of freedom. The coordinate space used to define the probe’s pose may also be referred to as the probe space.

[0087] In an embodiment, the robotic system determines the z-position using a sensor, e.g., a pressure sensor or optical sensor, to maintain consistent contact with the patient’s body, while actively controlling only two translational degrees of freedom for moving the ultrasound probe. Assuming three rotational degrees of freedom, that results in a so-called 5D-system, and a 5D probe space. The computer system 102 receives the plurality of ultrasound frames captured by the ultrasound probe at the respective probe pose, as indicated in step 112. Optionally, the computer system may pre-process 128 the received ultrasound frames performing operations such as cropping, resizing, normalization, speckle reduction, smoothing, etc. Such pre-processing may facilitate the quality assessment described below. Each ultrasound frame is associated with a respective ultrasound probe pose. To that end, the computer system may receive information about the ultrasound probe’s pose from, e.g., the robotic controller. This information may be obtained from sensors, e.g. motion sensors, in the robotic system. Alternatively, the computer system can keep track of the (relative) ultrasound probe’s pose, based on provided movement control commands. In general, sensor information provided by the robotic system is more accurate, but depending on the robotic system, such sensor information may not be available.

[0088] In one embodiment, the received ultrasound frames may be captured along different scanning windows and axes to obtain a complete transthoracic echocardiographic (TTE) examination. For this purpose, the ultrasound frames received by the computer system may include 2D ultrasound images for various views, such as the parasternal long-axis (PLAX), parasternal short-axis (PSAX), apical, suprasternal and subcostal views, as well as Doppler images to assess blood flow dynamics and cardiac function. Especially in such embodiments, each ultrasound frame may be associated with a cardiac phase and / or with a respiratory phase.

[0089] In another embodiment, the ultrasound frames received by the computer system may correspond to images suited for other diagnostic applications, including 2D, 3D, or Doppler ultrasound images for scenarios such as foetal monitoring, vascular imaging, or abdominal imaging of organs such as the liver, kidneys, pancreas, or gallbladder.

[0090] The computer system may assess the quality of the ultrasound frames, e.g., by determining a quality score for each of the one or more received ultrasound frames, as shown in step 114. The quality score may indicate, comprise, or be based on a likelihood that the ultrasound frame corresponds to a target anatomical view and may represent a stopping criterion for determining when to stop capturing ultrasound frames. Also, the quality score may reflect whether the received ultrasound frame is suitable for diagnostic purposes, and / or whether other ultrasound frames obtained at the same probe pose may be suitable for diagnostic purposes; this can be especially relevant for dynamic scanning targets such as the heart, where the target region during the recording of the (clinical) ultrasound images may be different (e.g., in a different cardiac and / or respiratory phase, at least part of the time) than during the determination of the recording pose. Other stopping criteria 116 for capturing ultrasound frames may include: an iteration limit, defining a maximum number of captured ultrasound frames; a time limit, defining a maximum duration for capturing ultrasound frames; and / or an ultrasound image quality threshold.

[0091] Capturing ultrasound frames is performed using at least an image-based search algorithm to control the pose of the ultrasound probe 118. The image-based search algorithm may use the quality scores and / or the ultrasound frames as input to determine a change of the ultrasound probe pose (e.g., causing the ultrasound probe to move in a certain direction, to a certain location, or to a certain pose). Upon completion of capturing ultrasound frames (i.e. when the stopping criterion is satisfied 116), a recording pose is determined based on criteria that include at least the ultrasound image quality score. Optionally, additional sensor data, such as cardiac phase information derived from an ECG signal or breathing data, may also be used to refine the determination of the recording pose 120. In other embodiments, such cardiac and / or respiratory phase data may be derived from the ultrasound frames, either explicitly or implicitly. This recording pose is associated with the ultrasound probe pose at the time the ultrasound frame was captured. The diagnostic suitability may represent the accurate anatomical representation within the received ultrasound frame, ensuring that target anatomical features are clearly visible and accurately positioned within the image to support precise clinical interpretation. This accurate representation may entail capturing distinct boundaries, textures, and depths of relevant tissues or organs, providing high-quality visual data that enables detailed assessment of anatomical and physiological conditions. This accurate representation may include an implicit or explicit prediction of, or taking into account of, different cardiac and / or respiratory phases. For example, a quality score associated with a certain cardiac phase may be weighted to account for an expected quality for different cardiac phases; or a machine-learning model trained to determine a quality score may take cardiac phase implicitly (by deriving the cardiac phase from the ultrasound frame) or explicitly (by being provided with a signal representative of the cardiac phase) into account when determining the quality score for a given ultrasound frame.

[0092] In one embodiment, the quality assessment may be performed by a trained machine learning algorithm that has been trained to identify a target anatomical view of the input ultrasound frame. The target anatomical view might represent a specific orientation or crosssection of an anatomical region captured by the ultrasound probe, such as the parasternal long-axis or short-axis view in cardiac imaging, or specific organ views like those of the liver, kidneys, or pancreas. In some embodiments, the quality score may be determined using a machine learning algorithm in conjunction with traditional image processing methods, assessing factors such as image brightness, noise ratios, and other quality metrics.

[0093] The machine learning algorithm for assessing the ultrasound image quality may be trained in advance using a dataset of labelled 2D ultrasound images, where each labelled 2D ultrasound image is associated with a target label representing a target anatomical view, such as the parasternal long-axis, parasternal short-axis, apical, subcostal, suprasternal views or ultrasound images capturing specific organs like the liver, kidneys, or pancreas. The machine learning algorithm is trained to learn distinguishing features of each target anatomical view by analysing visual patterns within the images, including key structures, brightness, and contrast. This enables the machine learning algorithm to accurately identify the target anatomical view by recognizing consistent patterns that differentiate structural and positional characteristics in the received ultrasound frame. Using a machine learning algorithm, such as a convolutional neural network (CNN), the machine learning algorithm iteratively adjusts its parameters to minimize errors in predicting the correct anatomical view class. The output of the trained model is typically a probability score that indicates the likelihood of an ultrasound frame corresponding to a specific target anatomical view. This probability score can serve as a quality score, which the system can use to adjust the ultrasound probe pose during the ultrasound image capture process.

[0094] The computer system 102 may employ one or more search algorithms to generate movement control commands based on the determined quality scores for the received ultrasound images. Optionally, the movement control commands 126 may be sent to the robot controller, the robot controller being responsible for moving the ultrasound probe to a new recording pose 118. The search algorithms may employ various deterministic and non- deterministic movement patterns within a multi-dimensional (e.g., 5D or 6D) space corresponding to the ultrasound probe movement space. As is explained in more detail below with reference to Fig. 3 and 4, each search algorithm may be associated with one or more stopping criteria that may need to be met, either in order to transition to a next search algorithm or to start the ultrasound image capture. Depending on the implementations, search algorithms with more than one stopping criterion may require at least one criterion to be met, all criteria to be met, or one of several combinations of one or more criteria to be met. The one or more search algorithms may be iterative and / or multi-staged, and may be configured to visit, or explore, in at least one iteration or stage (but more typically in many or even all iterations or stages) points in probe space that were not visited in a previous iteration or stage. Thus, the search algorithm is not limited to a predetermined search path. This is particularly relevant for a scan in a anatomically challenging target region, such as for cardiac scans.

[0095] The computer system 102 can store the ultrasound probe poses and their associated quality scores throughout the ultrasound probe’s movement. Optionally the computer system may also store the captured ultrasound images. Storing the ultrasound probe poses and their associated quality scores enables the computer system to track and analyse the ultrasound probe movement in relation to image quality over time, allowing it to identify acceptable or even optimal poses for capturing high-quality diagnostic ultrasound images. Additionally, by storing the captured ultrasound images, the computer system can provide a reference dataset that aids in reviewing and verifying ultrasound image quality, at a particular ultrasound probe pose.

[0096] Once at least one ultrasound frame with a quality score exceeding a predefined quality threshold is recorded 116, and / or an accepted combination of one or more stopping criteria for the search algorithms are satisfied, the computer system 102 identifies a recording pose 120. If no ultrasound image has been recorded that meets the predefined quality score threshold, patient repositioning may be required. The computer system selects the recording pose by analysing the stored information about ultrasound image quality of the previously recorded and stored ultrasound probe poses. It may also incorporate additional available parameters, such as sensor data, to improve selection accuracy. For instance, in a TTE exam, the computer system could use an ECG signal (e.g., from a dedicated sensor, or derived from the captured ultrasound frames) to synchronize the ultrasound capture with specific phases of the cardiac cycle and / or the breathing rhythm, ensuring more consistent and diagnostically valuable images. Furthermore, the quality score may be correlated with particular phases of the cardiac cycle and / or the breathing rhythm, such that the highest quality score does not necessarily indicate the best recording pose but rather reflects the optimal image quality achievable at a given cardiac and / or respiratory phase. By combining quality scores and supplementary data, such as ECG-derived cardiac and / or breathing phase information, the computer system can identify and select the ultrasound probe pose from the stored set that is most likely to yield an optimal recording for diagnostic purposes.

[0097] Once the recording pose has been determined, the computer system 102 sends movement control signals 126 to the robot system to move the ultrasound probe to the identified recording pose, and to capture of one or more optimal ultrasound images for a specific anatomical view 122. In some embodiments, the control signals might be sent to the robot controller, the robot controller being responsible for moving the ultrasound probe to the identified optimal recording pose, while the recording control commands are sent to an ultrasound controller controlling the imaging and recording parameters of the ultrasound probe 106.

[0098] Optionally, the robotic ultrasound imaging system 100 may optimize the image quality of the optimal recorded ultrasound image by adjusting parameters of the ultrasound probe 106 such as gain, dynamic range, depth, and scan width, e.g., based on real-time image analysis. Image processing techniques may evaluate aspects of the recorded images, such as brightness, contrast, and clarity, and make the necessary adjustments to these parameters to improve the quality of the images. This autonomous adjustment can enhance image quality continuously throughout the scan, reducing the need for operator input and ensuring more consistent, high-quality images.

[0099] Thus, the robotic system illustrated in Fig. 1 enables autonomous recording of one or more ultrasound images during an ultrasound procedure. By employing a computer- implemented method that assesses image quality, generates movement commands, and, optionally, optimizes ultrasound probe parameters, the robotic ultrasound imaging system can efficiently capture high-quality images of specific anatomical views without manual intervention. This approach ensures that diagnostic images meet predefined quality standards, supporting accurate and consistent ultrasound imaging.

[0100] Fig. 2 depicts a flow-diagram of a computer-implemented method for autonomously recording one or more ultrasound images during an ultrasound procedure according to an embodiment. The method may comprise controlling 200 a robotic system to move an ultrasound probe relative to a patient’s body. This movement may be adjusted dynamically to ensure adequate scanning of relevant anatomical regions. As the ultrasound probe moves, a plurality of ultrasound frames is received 202, each of the plurality of ultrasound frames being associated with a specific ultrasound probe pose. The probe pose may be defined in a probe space. If the ultrasound procedure is a cardiac ultrasound procedure, each of the plurality of ultrasound frames may be associated with a respective cardiac and / or respiratory phase.

[0101] The method further comprises determining 204 a quality score of each ultrasound frame by calculating a quality score. The quality score may indicate a diagnostic suitability of the ultrasound frame, and / or of other ultrasound frames obtained at the same ultrasound probe pose (but with, possibly, a different cardiac and / or respiratory phase). In an embodiment, the quality score represents the probability that the received ultrasound frame corresponds to a target anatomical view. In some embodiments, one or more image parameters, such as brightness, contrast, and visibility of anatomical features, may be considered, additionally or alternatively, in deriving the quality score. In some embodiments, the quality score is further determined using additional sensor indicative of physiological signals (i.e., signals representative of a physiological state of the subject being scanned). For example, the additional sensor data may comprise ECG data, which is indicative of the cardiac and / or respiratory phase.

[0102] The method may further comprise controlling 205 a robotic system to move the ultrasound probe relative to the patient’s body, based on a search algorithm. The search algorithm may be an image-based search algorithm using as input at least one of the determined quality scores and the plurality of ultrasound frames. Optionally, the search algorithm may also use the respective ultrasound probe poses as input.

[0103] The method further comprises identifying 206 a recording pose by selecting an ultrasound probe pose through analysis of the stored information about the quality scores determined for the ultrasound frames associated with the (previously recorded and stored) ultrasound probe poses. Additionally, the selection of a recording pose may incorporate additional available parameters, such as sensor data, to improve selection accuracy. The computer system then controls 208 the robotic system to move the ultrasound probe to the identified recording pose, e.g., by sending commands to a robotic controller of the robotic system to move the probe to the identified recording pose. Once the ultrasound probe is positioned at the identified recording pose, the system captures and records 210 one or more ultrasound images. In particular in the context of a cardiac procedure, the one or more ultrasound images may comprise one or more ultrasound frames associated with one or more further cardiac and / or respiratory phases. While the computer system aims to optimize ultrasound image quality, the computer system typically prioritizes ensuring that sufficient ultrasound images are available for clinical evaluation.

[0104] Fig. 3 represents a flowchart illustrating a method for controlling a robotic system to move an ultrasound probe and to record an ultrasound image, according to an embodiment. The method employs one or more search algorithms, selected from a collection including: a direct search, a deterministic search algorithm, an adaptive search algorithm and a featurebased algorithm. As shown in the flowchart, the method may begin with the robotic system selecting and initiating a initial search algorithm, e.g., a direct search or a deterministic search algorithm 300. Either before or after the initial search algorithm is initiated, the robotic system may be instructed to make contact with the patient, e.g., using feedback from a force sensor or optical sensor to confirm contact. In other embodiments, an initial placement of the ultrasound probe on the patient’s body may be performed manually by an operator.

[0105] Following this, in step 302, the robotic system executes movement commands based on the selected search algorithm to move the ultrasound probe over the patient’s body and capture one or more preliminary ultrasound frames.

[0106] Once one or more preliminary ultrasound frames are captured, the robotic system may evaluate whether one or more specific stopping criteria for the initial search algorithm have been satisfied, as described in step 304. These criteria may include factors such as ultrasound image quality, anatomical feature visibility, an iteration limit or a time limit (as further described in Fig. 1). If the stopping criteria are not satisfied, the robotic system continues to apply the current search algorithm, iteratively moving the ultrasound probe pose until the stopping criteria are met. In some embodiments, the robotic system may complete the search algorithm, e.g., by performing a complete movement pattern, regardless of image- related factors such as ultrasound image quality and / or anatomical feature visibility.

[0107] If the stopping criteria are met for the current search algorithm, the robotic system then assesses the availability of additional search algorithms, as described in step 306. If additional search algorithms are available, the robotic system transitions to the next search algorithm 308 to further refine the ultrasound probe’s positioning and ultrasound image quality. This further refinement may include exploring previously unvisited points in probe space. In other embodiments, a next search algorithm is only initiated if no ultrasound frame with sufficient quality has been obtained so far. The decision whether or not to continue with a next search algorithm may depend on the image quality and / or the search algorithm(s) that already has or have been used. This iterative process of switching algorithms may continue until either the stopping criteria are satisfied and no additional algorithms are available 310 or an iteration limit was reached 320 without capturing an ultrasound frame that meets the stopping criterion of one of the search algorithms.

[0108] In an embodiment, if no satisfactory ultrasound frame is captured, the robotic system may prompt for patient repositioning 322. This prompt can alert an operator to adjust the patient’s position or orientation, ensuring that the robotic system can properly reach the target anatomy. Once repositioned, the robotic ultrasound imaging system can reinitiate the search algorithm for recording an ultrasound image.

[0109] In an embodiment, the robotic ultrasound imaging system stores the ultrasound probe poses, the quality of the captured ultrasound images, and optionally, the ultrasound images that meet at least a minimum image quality for each of the search algorithms in memory or on a drive. In another embodiment, the robotic ultrasound imaging system stores all the ultrasound probe poses, the quality of the captured ultrasound images, and the ultrasound images during the ultrasound probe movement, independent of the determined image quality.

[0110] Subsequently, in step 312 of Fig. 3, the system identifies an optimal ultrasound recording pose, by selecting an ultrasound probe pose from the stored ultrasound probe poses, based on the associated quality scores and, optionally, additional data such as sensor inputs or recorded timestamps. This selection process ensures that the selected ultrasound probe pose meets specific standards for diagnostic reliability. Following this, in an optional step 314, the robotic ultrasound imaging system may automatically optimize probe settings — such as gain, depth, and focus — to enhance image quality at the selected optimal recording pose. Finally, in block 316, the robotic system captures and records the ultrasound image at the optimised pose. This automated process allows the robotic ultrasound system to achieve high-quality, consistent imaging through image-based search algorithms and realtime adjustments, reducing the need for manual intervention and ensuring reliable diagnostic outcomes. Depending on the ultrasound procedure several recording poses may be required. For subsequent recording poses that do not require patient repositioning, the system may use previously captured data to, e.g., skip one or more search algorithms and / or to estimate a good starting pose for the ultrasound probe.

[0111] Fig. 4 presents a flowchart of a method using a collection of search algorithms according to an embodiment. The method may be employed by the robotic ultrasound imaging system to search and record an ultrasound image as outlined in Fig. 3. Fig. 4 emphasizes the interaction between the search algorithms and the decision-making process within the robotic ultrasound imaging system for autonomously determining an ultrasound probe pose to record an ultrasound image of an optimal quality.

[0112] The search process initiates with determining whether the relative location of the ultrasound probe relative to the patient’s body is known, as indicated in step 400. If this location is known, for example from a previous ultrasound scan of the same patient, the system proceeds directly to the next steps. If the location is unknown, the system assesses whether a direct search is allowed, as shown in step 402. In an embodiment, direct search may be allowed at the start of the ultrasound examination if it has not been previously employed during the current search process, or if its previous use resulted in recorded ultrasound images that exceeded a predefined quality threshold.

[0113] In one embodiment, the direct search algorithm may be initiated at the start of the ultrasound recording process 404. The direct search algorithm may utilize a navigation machine-learning model, which analyses the captured ultrasound frames to estimate the x and y direction needed to locate a target body part (e.g., the heart, other organs). This search algorithm is designed to perform coarse adjustments, enabling the robotic system to efficiently approximate the location of the target body part, even if the resulting ultrasound frames do not capture the detailed anatomical features of the target body part. The model used in the direct search algorithm may fail to locate the target body part in cases of low signal-to-noise ratio (SNR). In such cases, the system may be configured to handle such failure scenarios by transitioning from the direct search algorithm to another search algorithm.

[0114] If a direct search is allowed, the system proceeds to execute a direct search algorithm, as shown in step 404, to position the ultrasound probe and capture ultrasound frames. Following this, the robotic system may assess the quality of the acquired ultrasound frames in step 406 to determine if they correspond to the target anatomical view and whether they meet sufficient diagnostic standards. In one embodiment, the ultrasound image quality score is evaluated against a predetermined quality score threshold. If the ultrasound image satisfies a stopping criterion such as meeting or exceeding a quality score threshold, and an iteration or time limit is not exceeded, the system continues with the subsequent adaptive search algorithm 412; if not, the search process may continue with a deterministic search algorithm 408. In another embodiment, the system may continue in both scenarios with a deterministic search algorithm.

[0115] In some embodiments (e.g., as described relative to Fig. 1), a trained machine-learning model may assess ultrasound frame quality by outputting a probability score that reflects the likelihood of the captured ultrasound image accurately representing the target anatomical view. In some embodiments, the quality score may also be determined using a machine learning algorithm in conjunction with traditional image processing methods, assessing factors such as image brightness, noise ratios, anatomical feature visibility.

[0116] If the direct search algorithm is not allowed, or if employing the direct search algorithm does not yield an ultrasound frame with adequate quality (the quality score is not exceeding a predefined threshold), the robotic system initiates a deterministic search algorithm, outlined in step 408.

[0117] In one embodiment, the deterministic search algorithm adjusts the ultrasound probe’s pose according to a predefined movement pattern, which may include a spiral or lawnmowerlike path. For example, a spiral movement pattern may be used to scan parasternal views, while a lawnmower-like pattern may be applied for apical and subcostal views during a transthoracic echocardiogram. Throughout this process, the algorithm may maintain consistent values for yaw, pitch, and roll to ensure stability and patient comfort while covering a broad area of the scanned body part.

[0118] During the deterministic search algorithm, the system evaluates a stopping criterion, which may include one or more of: assessing the quality, evaluating a signal-to-noise-ratio, checking the ultrasound probe pose against the robot arm limits, checking an iteration or time limit, of the received ultrasound frames in step 410. In one embodiment, if at least one captured ultrasound frame satisfies the required stopping criterion the process advances to perform an adaptive search algorithm 412. This transition may occur immediately or after the deterministic search has completed (i.e. when an iteration or time limit has been reached and at least one ultrasound frame satisfying the stopping criterion has been received). If no images that satisfy the stopping criterion are received within a predetermined amount of time or a predetermined number of iterations the robotic system prompts repositioning of the patient.

[0119] The robotic system may initiate an adaptive search algorithm, as shown in step 412, either when the captured ultrasound frames satisfy the stopping criteria) following the deterministic search algorithm or when the ultrasound probe’s pose relative to the patient’s body is known, as indicated in step 400. The adaptive search algorithm defines one or more second movement patterns for moving the ultrasound probe relative to the patient’s body, the one or more second movement patterns comprising movements along multiple degrees of freedom. The one or more second movement patterns may be dynamically adjusted based on the quality scores determined for the plurality of ultrasound frames obtained during execution of the adaptive search algorithm.

[0120] As previously mentioned, the adaptive search algorithm operates in conjunction with an ultrasound frame quality score 414. In one embodiment, the adaptive search algorithm optimizes each of the five or six (e.g., (x, y, a, p, y) or (x, y, z, a, p, y)) axes independently by moving the ultrasound probe incrementally in positive and negative directions along each axis (e.g., for the X axis, moving left and right) to find the pose of the ultrasound probe that corresponds to an ultrasound frame with the highest frame quality score. Once the best pose on a given axis is identified, the adaptive search algorithm adjusts the ultrasound probe pose accordingly and proceeds to optimizing the next axis. After completing an iteration across all five or six axes (as mentioned before, z may be kept constant or controlled independently to ensure good patient contact), the adaptive search algorithm moves the ultrasound probe to the optimal probe pose identified so far. In an embodiment, the adaptive search algorithm may reduces the exploration distance for the next iteration, refining its search range.

[0121] In an embodiment, the adaptive search algorithm may optionally transition to a featurebased search algorithm 420 before recording one or more ultrasound images 418. This transition to a feature-based search algorithm may be optionally enabled if a predefined number of high-quality frames are captured during the movement of the ultrasound probe under the adaptive search algorithm. If no transition occurs, the adaptive search algorithm may continue until it reaches an iteration limit or time limit.

[0122] Finally, when the iteration or time limit is reached, the ultrasound probe pose is adjusted to an identified recording pose 416. This pose is determined based on previously recorded probe poses during the ultrasound probe movement and their associated quality scores, with optional input from other available sensors, such as an ECG signal, as discussed previously with respect to Fig. 1.

[0123] Once an optimal pose is established, the system proceeds to step 418 to capture and record one or more ultrasound images.

[0124] Fig. 4 provides an in-depth illustration of the sequence and interaction of search algorithms — direct, deterministic, adaptive, feature-based — that collectively enable the robotic ultrasound system to autonomously locate, refine, and record high-quality diagnostic ultrasound images with minimal or no manual intervention. The structured approach also ensures that, if no satisfactory ultrasound images are obtained, patient repositioning may be prompted to optimise results. Other embodiments may use more, fewer, or different search algorithms, or different decision criteria to move from one search algorithm to another.

[0125] The optional feature-based search algorithm 420 may define one or more third movement patterns for moving the ultrasound probe relative to the patient’s body. The one or more third movement patterns may be determined based on one or more localised anatomical features in the plurality of ultrasound frames. The one or more localised anatomical features being determined using a feature-detection algorithm.

[0126] The feature-based search algorithm is employed in conjunction with an image quality score and a feature-detection algorithm to further improve capturing high-quality ultrasound images that contain the relevant anatomical target features within the robotic ultrasound imaging system. The quality model assigns a quality score to each ultrasound frame encountered during the adaptive search algorithm. The feature-detection algorithm identifies and localises specific anatomical features within the ultrasound frame for each ultrasound frame having a quality score exceeding a predetermined quality score threshold. The identified anatomical features may be further mapped to a 3D location in the (x, y, z) space by correlating it with the robot’s ultrasound probe pose at the time of capture.

[0127] The feature-based search algorithm leverages anatomical information gathered during the search (such as 3D location of heart valves and walls) to enhance the robotic system’s ability to record high-quality ultrasound images. By constructing a partial 3D map of the scanned body part (e.g., the heart), the feature-based search algorithm enables the system to virtually explore and “slice” this 3D map from different angles, optimizing search pathways that have not been previously attempted during previous search algorithms, in this example the direct, deterministic or adaptive search algorithms.

[0128] The feature-based search algorithm generates a location in the 3D space (x, y, z) per movement cycle (with each cycle involving the optimization of a single axis separately) by optimizing for various costs and rewards. Each 3D location generated by the feature-based search algorithm is initially selected from stored ultrasound probe locations during the adaptive search algorithm. These initial locations may be further refined using an optimisation process, typically by minimising or maximising a cost. The optimised cost may encompass several factors: e.g., minimising a distance to a median location of the best observed anatomical features in the ultrasound frame, maximising a distance from previously explored ultrasound probe poses, improving an average quality score of neighbouring probe poses relative to the current ultrasound probe pose, and conducting the feature-based search within specific boundaries around the current probe pose (e.g., within a 15 mm radius and 15° angle of the current optimal ultrasound probe pose).

[0129] The feature-detection algorithm may include a machine-learning model such as an object-detection algorithm or an instance-segmentation algorithm. To that end, the machinelearning model may include one or more deep neural networks. The machine-learning model may be trained using an input dataset comprising ultrasound images and corresponding target locations for the one or more localized anatomical features in each ultrasound image. The feature-detection algorithm may output a predicted location and / or a confidence score for each of the one or more localized anatomical features.

[0130] Fig. 5 illustrates a method for autonomously recording one or more ultrasound images during an ultrasound procedure within a robotic ultrasound imaging system according to an embodiment. It includes an example display of a user interface that may be used for the analysis and debugging of the automatic recording system. Fig. 5 depicts the method steps performed by the computer system after recording the one or more ultrasound images. Each dot in the figure 502 represents an individual ultrasound frame recorded during the ultrasound probe’s movement according to one of the search algorithms described in Fig. 3 and Fig. 4. The colour gradient of the dots indicates variations in the ultrasound frame quality score.

[0131] The timeline at the bottom displays transitions between different search algorithms 504, with each tab representing a specific search algorithm used in the ultrasound probe’s positioning process. For example, the robotic system moves through all the search algorithms such as direct search algorithm, deterministic, adaptive search algorithms and feature-based search algorithm, each represented in the timeline 504. In this example, the sequential progression through these search algorithms reflects the robotic system’s method to achieve an optimal probe pose for optimal, high-quality recording of an ultrasound image. Optionally, upon finalizing the search process, the robotic system may further optimise the captured ultrasound image 506 to enhance its quality. This optimisation may involve adjusting parameters such as gain, dynamic range, dept, zoom, etc. Following this optimisation step, the system proceeds to recording one or more ultrasound images 508 while the ultrasound probe is positioned at the identified recording pose.

[0132] Fig. 6 illustrates an example of the output of the feature-detection algorithm which can be used by the feature-based search algorithm within a robotic ultrasound imaging system. In this figure, an ultrasound frame is displayed, which can be considered a 2D slice of a 3D space.

[0133] The highlighted regions 602I-3 (e.g., mitral valve, aorta top, aorta bottom, etc.) represent localised anatomical features identified by the feature-detection algorithm, within the ultrasound frame. Each captured ultrasound frame may be associated with a specific pose of the ultrasound probe in the 3D space. As a result, the identified anatomical features in the ultrasound frame (in the 2D slice) can be mapped into the 3D space of the ultrasound probe. This mapping allows the feature-based search algorithm to utilize the anatomical features by slicing the 3D space (i.e., moving the ultrasound probe) based on the information about the identified anatomical features.

[0134] These localised anatomical features may be used by the feature-based search algorithm to further optimise the ultrasound image quality ensuring that relevant anatomical features are accurately captured, as further described in Fig. 4. Thus, the feature-based search algorithm allows the robotic system to continuously assess and adjust the ultrasound probe’s pose using feedback from the localised anatomical features to enhance ultrasound image quality and eventually to improve diagnostic reliability.

[0135] The localised anatomical features may also be used for autonomous non-B-mode image recording, as described below.

[0136] Fig. 7 illustrates a flow-diagram describing the steps of a computer-implemented method for autonomously recording and optimizing non-B-mode ultrasound images during an ultrasound procedure. The method starts by receiving a B-mode ultrasound image 700, generated by an ultrasound probe, the B-mode ultrasound image corresponding to a target anatomical view. A non-B-mode imaging modality and, optionally, one or more anatomical features, are either determined or received as input 702. Then one or more anatomical features are received or determined, a location for each of the one or more anatomical features within the B-mode ultrasound image may be determined or received 704. For example, the one or more anatomical features may be detected using a feature-detection algorithm, or the one or more anatomical features may be indicated by a user.

[0137] The method further comprises, in a step 706, determining a plurality of beam paths and / or volumes for the received non-B-mode imaging modality. The plurality of beam paths and / or volumes may be determined based on the location(s) of each of the one or more anatomical features. For each determined beam path and / or volume, a quality parameter is determined 708. A beam path and / or a volume is selected from the plurality of beam paths and / or volumes based on the determined quality parameter 710.

[0138] A step 712 comprises recording one or more non-B-mode ultrasound images using the non-B-mode imaging modality and the selected beam path and / or volume.

[0139] Determining the plurality of beam paths or volumes may be based on one or more of: a feature-based method, a flow-based method, or a pattern-based method. In a feature-based method, the beam paths or volumes are generated using one or more locations corresponding to one or more anatomical features identified as outputs from a feature-detection algorithm applied to the B-mode ultrasound image. In one embodiment, the one or more anatomical features are determined using a feature-detection algorithm, as described in more detail in Fig. 4 and Fig. 6. Each identified anatomical feature may be represented as a set of pixels or voxels, or as a delineated area. Consequently, the location of an anatomical feature may be expressed either as a collection of 2D or 3D coordinates (i.e., the x, y, z coordinates of each pixel or voxel associated with the anatomical feature, as shown in Fig. 8, 804i) or as coordinates defining the bounding area or volume surrounding the identified anatomical feature (Fig. 8, 8042). These 2D or 3D coordinates are defined relative to the ultrasound image space, and in one embodiment, they may also be mapped relative to the ultrasound probe pose space.

[0140] In another embodiment, a method for determining a plurality of beam paths and / or volumes may utilise a flow-based method. In this method, the beam paths and / or volumes are generated by sampling relevant regions in the B-model ultrasound image, the regions being identified based on flow velocity data. This flow velocity data may be derived from flow- related imaging modalities, such as colour flow mapping (CFM) of the received B-mode ultrasound image. For example, in a TTE exam, CFM enables the visualisation of blood flow direction and speed in the B-mode ultrasound image. By leveraging this data, the flow-based method identifies regions of interest within the B-mode ultrasound image that exhibit specific dynamic flow characteristics. The method then focuses on sampling the plurality of beam paths and / or volumes within these identified regions of interest.

[0141] Alternatively, in a pattern-based method, the beam paths or volumes are generated using a predetermined pattern around an initial estimate. The initial estimate may be derived from a machine-learning model.

[0142] The machine learning model may include one or more architectures, such as convolutional neural networks (CNNs) and / or transformer-based models designed for image analysis, including Vision Transformer-architectures.

[0143] This machine-learning model may be trained with input data consisting of B-mode ultrasound images paired with target beam paths and / or volumes that correspond to one or more non-B-mode ultrasound images. During training, the machine-learning model optimizes a loss function that calculates the error between the predicted beam paths and / or volumes generated by the model and the actual target beam paths and / or volumes from the non-B- mode ultrasound images. By minimizing this error, the machine-learning model learns to accurately predict optimal beam paths and / or volumes based on the anatomical information in the B-mode images.

[0144] In another embodiment, the machine learning model may incorporate a featuredetection algorithm, such as the one described in greater detail in relation to the featurebased search algorithm illustrated in Fig. 4 and Fig. 6 or the feature-based method for determining the plurality of beam paths and / or volumes illustrated in Fig. 7.

[0145] In one embodiment, the quality parameter used to select the optimal beam path and / or volume for non-B-mode ultrasound imaging is determined based on a measurement of flow velocity and / or the signal-to-noise ratio (SNR) within the non-B-mode ultrasound image. For example, in Doppler ultrasound imaging, the system may analyse the captured non-B-mode images to measure the flow velocity of blood through a vessel or valve. A higher flow-velocity measurement may indicate an accurately positioned beam path aligned with the target anatomy. Additionally, the system may calculate the signal-to-noise ratio within the non-B- mode image, where a higher SNR indicates a clearer image with less background noise. By using these parameters — flow velocity and SNR — the system can assess the quality of each beam path and / or volume, selecting those that yield the most diagnostically useful ultrasound images.

[0146] In another embodiment, the quality parameter is determined using a non-B-mode quality assessment model, which comprises a machine learning model configured to classify or predict (via regression) the quality parameter of a non-B-mode ultrasound image.

[0147] For example, the non-B-mode quality assessment model may be trained on ultrasound sweep data to produce masks that segment velocity waveforms and estimate parameters such as the signal-to-noise ratio (SNR) and / or the maximum flow velocity based on the segmented masks.

[0148] In one embodiment, recording one or more non-B-mode ultrasound images comprises a further non-B-mode image optimization process, wherein the non-B-mode image optimization comprises adjusting one or more imaging parameters such as sweep speed, baseline, sample volume, or gain, to improve the non-B-mode ultrasound image quality parameter and potentially to reduce aliasing of the non-B-mode ultrasound images.

[0149] Fig. 8 illustrates an example of a non-B-mode ultrasound image recording performed autonomously by a robotic ultrasound imaging system during an echocardiography exam, according to an embodiment.

[0150] The robotic ultrasound imaging system comprises a processor configured to execute the operations within the modules described in Fig. 8. In the example depicted in Fig. 8, the process begins by receiving a first B-mode ultrasound image. The first B-mode ultrasound image may be generated by an ultrasound probe of the robotic ultrasound imaging system, and can correspond to a target anatomical view. In Fig. 8 this operation is carried out by a B- mode ultrasound image capture module 802. In an embodiment, the B-mode ultrasound image capture module is configured to execute a method as described with reference to Figs. 1-5. As an example, the particular received B-mode ultrasound image depicted in Fig. 8 corresponds to an apical four chamber view of the heart.

[0151] The process continues with an anatomical feature-detection module 804, which receives or determines one or more anatomical features and a non-B-mode imaging modality, and determines the location of the one or more anatomical features, e.g., using a feature-detection algorithm. In an embodiment, the feature-detection algorithm is the same algorithm used for the feature-based search algorithm described above.

[0152] The feature-detection algorithm may include a machine-learning model such as an object detection algorithm or an instance segmentation algorithm. The machine-learning model may include one or more deep neural networks. The machine-learning model may be trained using an input dataset comprising ultrasound images and corresponding target locations for the one or more anatomical features in each ultrasound image. The featuredetection algorithm may output a predicted location and / or a confidence score for each of the one or more anatomical features.

[0153] In the particular example shown in Fig. 8, the identified anatomical feature is the tricuspid valve, represented by a cluster of points in the four-chamber view ultrasound image 804i. In the PLAX view 8042 example, the localized anatomical features are determined using an object detection algorithm which segments features such as the left ventricle, aorta and left atrium, displaying them in different shades of grey.

[0154] Following the identification of anatomical features in the B-mode ultrasound image, a beam paths / volumes determination module 806 determines one or more beam paths and / or volumes for non-B-mode imaging based on the identified anatomical features and the specified non-B-mode imaging modality. Each beam path or volume may then be assessed for measurement quality within the non-B-mode measurements / quality module 808 where quality parameters are calculated for each determined beam path and / or volume. In the example shown in module 808, Pulse Wave (PW) Doppler measurements are taken at the mitral valve (MV), with the E (1) and A (2) wave velocity measurements displayed. The highest MV inflow may be used as a quality parameter for selecting a beam path and / or volume from the plurality of beam paths and / or volumes for performing the non-B-mode measurements.

[0155] In an embodiment, a non-B-mode ultrasound image optimization module 810 may further optimize the non-B-mode measurements, e.g., by automatically adjusting parameters such as: gain, baseline, dynamic range, etc using traditional image processing techniques. These adjustments may include generating masks of the computed flow and applying iterative refinements to improve measurement accuracy.

[0156] Fig. 9 illustrates modules of a computer system 900 of a robotic ultrasound imaging system according to an embodiment. The computer system comprises a computer-readable program, that may comprise the modules shown in Fig. 9.

[0157] The computer system comprises a B-mode ultrasound image recording module 902, which records one or more B-mode ultrasound images of a target anatomical view. These ultrasound images may be evaluated by the B-mode ultrasound image quality module 904 to ensure they exceed a predefined quality score threshold. The B-mode ultrasound image recording module uses the B-mode ultrasound image quality module to perform a search using a collection of search algorithms for recording an optimal ultrasound image. The anatomical feature-detection module 906 determines anatomical features within the B-mode images, that may be used by the feature-based search algorithm to further improve the quality of the recorded ultrasound image.

[0158] The B-mode ultrasound image optimization module 908 may further refine the quality of the B-mode images by autonomously adjusting ultrasound probe parameters. The Non-B- mode ultrasound image recording module 910 receives a recorded B-mode ultrasound image and records non-B-mode images, such as Doppler (Colour Doppler, Pulse Wave Doppler, Continuous Wave Doppler), M-Mode or Tissue Doppler. In an embodiment, the non-B-mode ultrasound image recording is based on determining one or more beam paths and / or volumes using the location of each of the one or more anatomical features determined in the B-mode ultrasound image 906 and selecting a beam path and / or a volume from the plurality of beam paths and / or volumes based on a determined quality parameter 912.

[0159] In an embodiment, the non-B-mode ultrasound images may be further optimised by the non-B-mode ultrasound image optimization module 914, which adjusts parameters such as gain, baseline, and dynamic range to improve the measurements accuracy and reduce possible image aliasing.

[0160] Fig. 10 is a block diagram illustrating exemplary data processing systems described in this disclosure. Data processing system 1000 may include at least one processor 1002 coupled to memory elements 1004 through a system bus 1006. As such, the data processing system may store program code within memory elements 1004. Further, processor 1002 may execute the program code accessed from memory elements 1004 via system bus 1006. In one aspect, data processing system may be implemented as a computer that is suitable for storing and / or executing program code. It should be appreciated, however, that data processing system 1000 may be implemented in the form of any system including a processor and memory that is capable of performing the functions described within this specification.

[0161] Memory elements 1004 may include one or more physical memory devices such as, for example, local memory 1008 and one or more bulk storage devices 1010. Local memory may refer to random access memory or other non-persistent memory device(s) generally used during actual execution of the program code. A bulk storage device may be implemented as a hard drive or other persistent data storage device. The processing system 1000 may also include one or more cache memories (not shown) that provide temporary storage of at least some program code in order to reduce the number of times program code must be retrieved from bulk storage device 1010 during execution.

[0162] Input / output (I / O) devices depicted as input device 1012 and output device 1014 optionally can be coupled to the data processing system. Examples of input device may include, but are not limited to, for example, a keyboard, a pointing device such as a mouse, or the like. Examples of output device may include, but are not limited to, for example, a monitor or display, speakers, or the like. Input device and / or output device may be coupled to data processing system either directly or through intervening I / O controllers. A network adapter 1016 may also be coupled to data processing system to enable it to become coupled to other systems, computer systems, remote network devices, and / or remote storage devices through intervening private or public networks. The network adapter may comprise a data receiver for receiving data that is transmitted by said systems, devices and / or networks to said data and a data transmitter for transmitting data to said systems, devices and / or networks. Modems, cable modems, and Ethernet cards are examples of different types of network adapter that may be used with data processing system 1000. As pictured in FIG. 10, memory elements 1004 may store an application 1018. It should be appreciated that data processing system 1000 may further execute an operating system (not shown) that can facilitate execution of the application. Application, being implemented in the form of executable program code, can be executed by data processing system 1000, e.g., by processor 1002. Responsive to executing application, data processing system may be configured to perform one or more operations to be described herein in further detail.

[0163] In one aspect, for example, data processing system 1000 may represent a client data processing system. In that case, application 1018 may represent a client application that, when executed, configures data processing system 1000 to perform the various functions described herein with reference to a “client”. Examples of a client can include, but are not limited to, a personal computer, a portable computer, a mobile phone, or the like.

[0164] In another aspect, data processing system may represent a server. For example, data processing system may represent an (HTTP) server in which case application 1018, when executed, may configure data processing system to perform (HTTP) server operations. In another aspect, data processing system may represent a module, unit or function as referred to in this specification.

[0165] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. As used herein, the singular forms “a,” “an,” and “the” are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms “comprises” and / or “comprising,” when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0166] The corresponding structures, materials, acts, and equivalents of all means or step plus function elements in the claims below are intended to include any structure, material, or act for performing the function in combination with other claimed elements as specifically claimed. The description of the present invention has been presented for purposes of illustration and description, but is not intended to be exhaustive or limited to the invention in the form disclosed. Many modifications and variations will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the invention. The embodiment was chosen and described in order to best explain the principles of the invention and the practical application, and to enable others of ordinary skill in the art to understand the invention for various embodiments with various modifications as are suited to the particular use contemplated.

Claims

CLAIMS1. A computer-implemented method for autonomously recording one or more ultrasound images during an ultrasound procedure of a heart, the method comprising: receiving a plurality of ultrasound frames from an ultrasound probe, each of the plurality of ultrasound frames being associated with a respective ultrasound probe pose in a probe space and with a respective cardiac and / or respiratory phase; determining a quality score for each of the plurality of ultrasound frames, the quality score indicating diagnostic suitability of the ultrasound frame, wherein the quality score is representative of a likelihood that the ultrasound frame corresponds to a target anatomical view; controlling a robotic system to move the ultrasound probe in the probe space relative to a patient’s body, based on an image-based search algorithm using as input at least one of: the determined quality scores and the plurality of ultrasound frames, and, optionally, using as input the respective ultrasound probe poses; identifying a recording pose using the determined quality scores, the recording pose being a first respective ultrasound probe pose associated with a first ultrasound frame of the plurality of ultrasound frames; controlling the robotic system to move the ultrasound probe to the recording pose; and recording the one or more ultrasound images while the ultrasound probe is positioned at the identified recording pose, the one or more ultrasound images comprising one or more ultrasound frames associated with one or more further cardiac and / or respiratory phases.

2. The method as claimed in claim 1, wherein the image-based search algorithm is configured to at least explore probe poses in the probe space outside of a predetermined scanning path.

3. The method as claimed in claim 1 or 2, wherein each of the one or more ultrasound image comprises one or more frames.

4. The method as claimed in any of the preceding claims, wherein the quality score is further determined using additional sensor data, preferably the additional sensor data comprising physiological signals, more preferably the additional sensor data comprising ECG data.

5. The method as claimed in any of the preceding claims, wherein the ultrasound probe pose defines a location and / or an orientation.

6. The method as claimed in any of the preceding claims, wherein the quality score is based on an output from a machine learning algorithm, the machine learning algorithm being trained to either classify input ultrasound frames into a plurality of predefined anatomical views, or to regress a likelihood score indicating the degree to which an ultrasound framecorresponds to a target anatomical view; and wherein the output of the machine learning algorithm represents either a classification confidence score indicating the likelihood that an input ultrasound frame corresponds to one of the plurality of predefined anatomical views, or a regressed likelihood score.

7. The method as claimed in any of the preceding claims, wherein the image-based search algorithm being selected from a collection of search algorithms comprising one or more of: a direct search algorithm, an adaptive search algorithm, a feature-based search algorithm.

8. The method as claimed in claim 7, wherein: the direct search algorithm uses the one or more ultrasound frames of the plurality of ultrasound frames as input to a navigation machine-learning model to predict a target pose corresponding to the target anatomical view relative to the one or more respective ultrasound poses associated with the one or more ultrasound frames; and / or the adaptive search algorithm defines one or more second movement patterns for moving the ultrasound probe relative to the patient’s body, the one or more second movement patterns comprising movements along multiple degrees of freedom, the one or more second movement patterns being dynamically adjusted based on the quality scores determined for the plurality of ultrasound frames obtained during execution of the adaptive search algorithm; and / or the feature-based search algorithm defines one or more third movement patterns for moving the ultrasound probe relative to the patient’s body, the one or more third movement patterns being determined based on one or more localized anatomical features in the plurality of ultrasound frames, the one or more localized anatomical features being determined using a feature-detection algorithm.

9. The method as claimed in claim 8, wherein the feature-detection algorithm comprises a machine-learning model such as an object-detection algorithm or an instance-segmentation algorithm, preferably the machinelearning model including one or more deep neural networks, preferably the machine-learning model being trained using an input dataset comprising ultrasound frames and corresponding target locations for the one or more localized anatomical features in each ultrasound frame; and wherein the feature-detection algorithm outputs a predicted location and / or a confidence score for each of the one or more localized anatomical features.

10. The method as claimed in any of claims 7-9, where controlling the robotic system to move the ultrasound probe relative to the patient’s body comprises:moving the ultrasound probe according to a first search algorithm until a first stopping criterion is met, the first search algorithm either being selected from the collection of search algorithms or the first search algorithm being a deterministic search algorithm, preferably the deterministic search algorithm defining one or more first movement patterns for moving the ultrasound probe relative to the patient’s body, preferably the one or more first movement patterns comprising a spiral pattern and / or a lawnmower pattern; and moving the ultrasound probe according to a second search algorithm from the collection of search algorithms until a second stopping criterion is met.

11. The method according to any of claims 1-10, wherein the one or more ultrasound images comprise at least one first B-mode ultrasound image, the method further comprising: receiving the at least one B-mode ultrasound image, generated by an ultrasound probe, the B-mode ultrasound image corresponding to a target anatomical view; receiving or determining a non-B-mode imaging modality; determining a plurality of beam paths and / or volumes for the non-B-mode imaging modality; determining a quality parameter for each determined beam path and / or volume; selecting a beam path and / or volume based on the quality parameter and based on the non-B-mode imaging modality; and recording one or more non-B-mode ultrasound images using the non-B-mode imaging modality and the selected beam path and / or volume.

12. A computer-implemented method for autonomously recording a non-B-mode ultrasound image during an ultrasound procedure of a heart, the method comprising: receiving at least one B-mode ultrasound image, generated by an ultrasound probe, the B-mode ultrasound image corresponding to a target anatomical view; receiving or determining a non-B-mode imaging modality; determining a plurality of beam paths and / or volumes for the non-B-mode imaging modality; determining a quality parameter for each determined beam path and / or volume; selecting a beam path and / or volume based on the quality parameter and based on the non-B-mode imaging modality; and recording one or more non-B-mode ultrasound images using the non-B-mode imaging modality and the selected beam path and / or volume.

13. The method as claimed in claim 12, wherein determining a plurality of beam paths and / or volumes is based on one or more of the following: a feature-based method, wherein the plurality of beam paths and / or volumes are generated based on one or more locations of one or more anatomical features derived as output from a feature-detection algorithm applied to the B-mode ultrasound image; a flow-based method, wherein the plurality of beam paths and / or volumes are generated by sampling regions in the B-mode ultrasound image, the regions being identified based on flow velocity data; ora pattern-based method, wherein the plurality of beam paths and / or volumes are generated using a predetermined pattern around an initial estimate.

14. The method as claimed in claim 13, wherein the feature-detection algorithm comprises a machine-learning model such as an object-detection algorithm or an instancesegmentation algorithm, preferably the machine-learning model including one or more deep neural networks, preferably the machine-learning model being trained using an input dataset comprising ultrasound frames and corresponding target locations for the one or more anatomical features in each ultrasound frame, and wherein the feature-detection algorithm outputs a predicted location and / or a confidence score for each of the one or more anatomical features.

15. The method as claimed in any of the claims 12-14, wherein the initial estimate of the pattern-based algorithm is determined using at least one of: the feature-detection algorithm; and a machine-learning model comprising one or more architectures, such as convolutional neural networks and / or transformer-based architectures, wherein the machine-learning model is trained using as input B-mode ultrasound images and a plurality of target beam paths and / or volumes corresponding to one or more non-B-mode ultrasound images, to optimize a loss function that computes an error between an output of the machine-learning model, the output representing a plurality of predicted beam paths and / or volumes, and the corresponding target beam paths and / or volumes of the non-B-mode ultrasound images.

16. The method as claimed in any one of the claims 12-15, wherein the quality parameter is derived from at least one of: a measurement of a flow velocity, a signal-to-noise ratio determined in the non-B-mode ultrasound image, or a non-B-mode quality-assessment model comprising a machine-learning model configured to classify or regress a non-B-mode ultrasound image quality score17. The method as claimed in any of the claims 12-16, wherein recording one or more non-B-mode ultrasound images comprises a non-B-mode image-optimization process, wherein the non-B-mode image optimization comprises optimising one or more imaging parameters such as sweep speed, baseline, sample volume, gain, dynamic range or scale.

18. The method according to any of claims 1-10, wherein the one or more ultrasound images comprise at least one first B-mode ultrasound image, the method further comprising the method according to any of claims 12-17, wherein the received at least one B-mode ultrasound image is the at least one first B-mode ultrasound image.

19. A robotic ultrasound imaging system comprising:a robotic system configured to move an ultrasound probe relative to a patient’s body; and a computer system comprising a computer-readable storage medium storing computer- readable program code and a processor, preferably a microprocessor, coupled to the computer-readable storage medium, wherein responsive to executing the computer-readable program code, the processor is configured to perform executable operations comprising: receiving a plurality of ultrasound frames from an ultrasound probe, each of the plurality of ultrasound frames being associated with a respective ultrasound probe pose in a probe space and with a respective cardiac and / or respiratory phase; determining a quality score for each of the plurality of ultrasound frames, the quality score indicating diagnostic suitability of the ultrasound frame, wherein the quality score is representative of a likelihood that the ultrasound frame corresponds to a target anatomical view; controlling a robotic system to move the ultrasound probe in the probe space relative to a patient’s body, based on an image-based search algorithm using as input at least one of: the determined quality scores and the plurality of ultrasound frames, and, optionally, using as input the respective ultrasound probe poses; identifying a recording pose using the determined quality scores, the recording pose being a first respective ultrasound probe pose associated with a first ultrasound frame of the plurality of ultrasound frames; controlling the robotic system to move the ultrasound probe to the recording pose; and recording the one or more ultrasound images while the ultrasound probe is positioned at the identified recording pose, the one or more ultrasound images comprising one or more ultrasound frames associated with one or more further cardiac and / or respiratory phases.

20. The robotic ultrasound imaging system as claimed in claim 19, wherein the one or more ultrasound images comprise a first B-mode ultrasound image, and wherein the processor is configured to perform further executable operations comprising: receiving the first B-mode ultrasound image, generated by the ultrasound probe, the B- mode ultrasound image corresponding to a target anatomical view; receiving or determining a non-B-mode imaging modality; determining a plurality of beam paths and / or volumes for the non-B-mode imaging modality; determining a quality parameter for each determined beam path and / or volume; selecting a beam path and / or a volume based on the quality parameter and based on the non-B-mode imaging modality; and recording one or more non-B-mode ultrasound images using the non-B-mode imaging modality and the selected beam path and / or volume.

21. A robotic ultrasound imaging system comprising: a robotic system configured to move an ultrasound probe relative to a patient’s body; anda computer system comprising a computer-readable storage medium storing computer- readable program code and a processor, preferably a microprocessor, coupled to the computer-readable storage medium, wherein responsive to executing the computer-readable program code, the processor is configured to perform executable operations comprising: receiving the first B-mode ultrasound image, generated by the ultrasound probe, the B- mode ultrasound image corresponding to a target anatomical view; receiving or determining a non-B-mode imaging modality; determining a plurality of beam paths and / or volumes for the non-B-mode imaging modality; determining a quality parameter for each determined beam path and / or volume; selecting a beam path and / or a volume based on the quality parameter and based on the non-B-mode imaging modality; and recording one or more non-B-mode ultrasound images using the non-B-mode imaging modality and the selected beam path and / or volume.

22. A computer program product comprising instructions which, when the computer program is executed by a computer, cause the computer to perform the method steps as claimed in any one of claims 1-18.

23. A non-transitory computer-readable storage medium comprising instructions which, when executed by a computer, cause the computer to perform the method steps as claimed in any one of claims 1-18.

Images