An ultrasound imaging system and method

Abstract

An ultrasound imaging system is provided for scanning a region of interest in a subject, the ultrasound imaging system comprising: an ultrasound probe; a manipulator arm to hold and move the ultrasound probe along a body surface of the subject; at least two image sensors, each image sensor using a different imaging modality to capture a different aspect of the subject for determining control data, wherein the at least two image sensors include a thermal sensor adapted to capture a thermal image to differentiate the body surface of the subject from the surrounding environment and a depth sensor adapted to capture one or more depth images to provide a three-dimensional view of the subject; and a controller coupled to the manipulator arm to cause the manipulator arm to move the ultrasound probe in accordance with the control data, wherein the control data includes location information to identify an initial position on the body surface of the subject for scanning the region of interest. A method of autonomously moving an ultrasound probe to an initial position on the body surface of the subject for scanning a region of interest in a subject and a method of conducting an ultrasound scan of a region of interest in a subject is also provided.

Description

An Ultrasound Imaging System and MethodTechnical Field

[0001] The present invention relates to an ultrasound imaging system and a method of ultrasound imaging, and more particularly an ultrasound imaging system and method for autonomously identifying an initial position on a body surface of a subject for ultrasound scanning a region of interest in the subject.Background of Invention

[0002] Ultrasound is an imaging technique that uses high-frequency (>20 kHz) sound waves to generate images of internal body structures. The applications of ultrasonography are diverse, with one of the most common applications being non-invasive diagnosis medical diagnosis. During a manual ultrasonography, a sonographer applies a conductive gel to the skin above the region of interest (ROI) and manipulates the ultrasound probe until a clear image of the region of interest is acquired. Ultrasound imaging offers several advantages over other diagnostic imaging techniques, such as magnetic resonance imaging (MRI) and computed tomography (CT), including affordability, speed and patient safety.

[0003] Manual ultrasound imaging is heavily reliant on the experience of the operator. People in remote and rural areas may experience restricted access to experienced medical professional and ultrasound technology, resulting in delayed diagnosis. Moreover, due to the prolonged and repetitive nature of manual sonography, a high incidence of work-related musculoskeletal disorders are reported amongst sonographers, including bursitis, muscle strain and peripheral nerve pathology, primarily affecting the upper neck and back.

[0004] Robotic systems for automating manual ultrasound procedures are being developed to offer a promising solution to both alleviate the stresses on operators, as well as addressing an increasing demand for remote imaging capabilities. However, existing robotic systems typically require manual intervention during the scanning process. The entire ultrasound scanning process encompasses tasks from localising the region of interest (ROI), placing the ultrasound probe on the skin surface above the ROI, and manipulating the ultrasound probe for ultrasound image acquisition. A notable limitation of current systems isthe reliance on human input for ROI localization, which impedes autonomous ultrasound imaging using robots.

[0005] To the best of our knowledge, there is currently no commercially implemented ultrasound imaging system available which enables a robot to autonomously identify an initial position on the body surface of the subject for conducting an ultrasound scan of a region of interest.

[0006] It would be desirable to provide a robotic system capable of autonomously identifying and moving an ultrasound probe to such a start point to reduce the stresses on sonographers locally, and to support remote ultrasound imaging operated by a remotely located sonographer that provides results comparable to an in-room scan.

[0007] A reference herein to a patent document or any other matter identified as prior art, is not to be taken as an admission that the document or other matter was known or that the information it contains was part of the common general knowledge as at the priority date of any of the claims.Summary of Invention

[0008] According to an aspect of the present invention, there is provided an ultrasound imaging system for scanning a region of interest in a subject, the ultrasound imaging system comprising: an ultrasound probe; a manipulator arm to hold and move the ultrasound probe along a body surface of the subject; at least two image sensors, each image sensor using a different imaging modality to capture a different aspect of the subject for determining control data, wherein the at least two image sensors include a thermal sensor adapted to capture a thermal image to differentiate the body surface of the subject from the surrounding environment and a depth sensor adapted to capture one or more depth images to provide a three-dimensional view of the subject; and a controller coupled to the manipulator arm to cause the manipulator arm to move the ultrasound probe in accordance with the control data, wherein the control data includes location information to identify an initial position on the body surface of the subject for scanning the region of interest.

[0009] In some embodiments, the control data further includes orientation information to identify an initial orientation of the ultrasound probe on the body surface of the subject for scanning the region of interest.

[0010] In a particular embodiment, the control data further includes trajectory information to enable autonomous movement of the ultrasound probe from a home position to the initial position and initial orientation for scanning of the region of interest.

[0011] In some embodiments, the thermal image and the one or more depth images captured by the image sensors are fused to form a fused depth-thermal composite image. The fused depth-thermal composite image may be used to determine the control data.

[0012] In certain embodiments, the thermal image and the one or more depth images are captured substantially simultaneously.

[0013] In some embodiments, the depth sensor comprises an RGB-D camera.

[0014] In certain embodiments, the manipulator arm is part of a collaborative robot.

[0015] According to another aspect of the present invention, there is provided a method of autonomously moving an ultrasound probe to an initial position on the body surface of the subject for scanning a region of interest in a subject, the method including the following steps: providing an ultrasound imaging system comprising an ultrasound probe, a manipulator arm to hold and move the ultrasound probe along a body surface of the subject, at least two image sensors, including a thermal sensor adapted to capture a thermal image to differentiate the body surface of the subject from the surrounding environment and a depth sensor adapted to capture one or more depth images to provide a three-dimensional view of the subject, and a controller coupled to the manipulator arm to cause the manipulator arm to move the ultrasound probe in accordance with a control data; acquiring at least a thermal image and one or more depth images; fusing the thermal image and the one or more depth images to determine the control data, wherein the control data includes location information to identify an initial position on the body surface of the subject for scanning the region of interest; and moving the ultrasound probe to the initial position on the body surface of the subject for scanning using the manipulator arm.

[0016] In some embodiments, the control data further includes orientation information to identify an initial orientation of the ultrasound probe on the body surface of the subject for scanning the region of interest.

[0017] In a particular embodiment, the control data further includes trajectory information to enable autonomous movement of the ultrasound probe from a home position to the initial position and initial orientation for scanning of the region of interest.

[0018] In another embodiment, the step of moving the ultrasound probe to the initial position on the body surface of the subject further includes managing a force exerted on the surface of the body of the subject by the ultrasound probe by providing torque and force feedback to the manipulator arm.

[0019] In some embodiments, the step of fusing the thermal image and the one or more depth images provides a fused depth-thermal composite image.

[0020] In a particular embodiment, the step of acquiring images using the thermal sensor and the depth sensor occurs substantially simultaneously.

[0021] In some embodiments, the depth sensor captures one or more RGB images using an RGB imaging module.

[0022] In certain embodiments, the step of fusing the thermal image with the one or more depth images to determine the control data further includes the following steps: estimating one or more parameters of the thermal sensor and the depth sensor; rectifying the thermal image and the RGB image using the estimated parameters to achieve alignment between the thermal image and the RGB image; and substituting the RGB image with the rectified thermal image; projecting one or more depth data points extracted from the one or more depth images onto the rectified thermal image using one or more known calibration parameters of the depth sensor to form a fused depth-thermal composite image comprising a three-dimensional point cloud including location and thermal intensity information.

[0023] In some embodiments, the step of estimating one or more parameters of the thermal sensor and the depth sensor comprises estimating intrinsic and extrinsic parameters of the thermal sensor and the RGB imaging module of the depth sensor.

[0024] In a particular embodiment, the step of rectifying the thermal image and the one or more RGB images is preceded by a preprocessing step.

[0025] The step of projecting one or more depth data points extracted from the one or more depth images onto the rectified thermal image may include using one or more known calibration parameters associated with the depth sensor to project depth image data acquired in the one or more depth images onto the rectified thermal image.

[0026] In certain embodiments, the control data further includes trajectory information to enable autonomous imaging of the region of interest.

[0027] According to still another aspect of the present invention, there is provided, a method of conducting an ultrasound scan of a region of interest in a subject, the method including the following steps: providing an ultrasound imaging system comprising an ultrasound probe, a manipulator arm to hold and move the ultrasound probe along a body surface of the subject, at least two image sensors, including a thermal sensor adapted to capture a thermal image to differentiate the body surface of the subject from the surrounding environment and a depth sensor adapted to capture one or more depth images to provide a three-dimensional view of the subject, and a controller coupled to the manipulator arm to cause the manipulator arm to move the ultrasound probe in accordance with a control data; acquiring at least a thermal image and one or more depth images; fusing the thermal image and the one or more depth images to determine the control data, wherein the control data includes location information to identify an initial position on the body surface of the subject for scanning the region of interest and orientation information to identify an initial orientation of the ultrasound probe on the body surface of the subject; autonomously moving the ultrasound probe from a home position to the initial position and the initial orientation using the manipulator arm based on trajectory information provided by the control data; establishing contact between the ultrasound probe and the surface of the body of the subject using the manipulator arm; managing a force exerted on the surface of the body of the subject by the ultrasound probe by providing torque and force feedback to the manipulator arm; scanning the region of interest in the subject to provide a plurality of ultrasound images of the region of interest; and moving the ultrasound probe upwards using the manipulator arm and subsequently returning the ultrasound probe to the home position.

[0028] In some embodiments, the step of fusing the thermal image and the one or more depth images to determine the control data comprises fusing the thermal image with the one or more depth images to provide a fused depth-thermal composite image.

[0029] In certain embodiments, the step of acquiring the thermal image and the one or more depth images occurs substantially simultaneously.

[0030] In a particular embodiment, the step of managing a force exerted on the surface of the body of the subject by the ultrasound probe comprises use of a force-torque control algorithm.

[0031] In the system and methods described above, the subject may be a human or an animal subject. Moreover, the region of interest may be one or more of a lumbar region, a cardiac region, an abdominal region, or a limb including an upper limb and / or a lower limb.Brief Description of Drawings

[0032] The invention will now be described in greater detail with reference to the accompanying drawings in which like features are represented by like numerals. It is to be understood that the embodiments shown are examples only and are not to be taken as limiting the scope of the invention as defined in the claims appended hereto.

[0033] Figure 1A shows an ultrasound imaging system according to an embodiment of the invention.

[0034] Figure IB shows the ultrasound imaging system of Figure 1A in more detail.

[0035] Figure 2 is a schematic of a communications network based on a Robot Operating System (ROS) to facilitate coordination between elements of the ultrasound imaging system according to an embodiment of the invention.

[0036] Figure 3 is a flowchart showing an overview of the depth-thermal sensor fusion approach according to an embodiment of the invention.

[0037] Figure 4A shows a disparity model and Figure 4B shows modelling of the disparity between between images captured by an RGB and a thermal camera when located a distance of 0.1 - 5m from the subject.

[0038] Figures 5A to 5D show a series of images of a subject's upper body lying prone on an ultrasound examination table captured using a) an RGB camera, b) a thermal camera, c) a fused depth-thermal mesh, and d) a segmented depth-thermal mesh.

[0039] Figures 6A to 6D show a series of images of a subject's upper body following preprocessing of the segmented depth-thermal image, showing a) the raw segmented depththermal mesh, b) the segmented depth-thermal mesh after applying Taubin filter, c) a point cloud generated from the mesh, and d) a point cloud after applying voxel down-sampling.

[0040] Figure 7 is a flowchart showing an overview of the template matching algorithm according to an embodiment of the invention.

[0041] Figures 8A to 8D show a series of images of a subject's upper body illustrating template alignment with the target point cloud, showing: a) the template point cloud with a selected point (sphere marked with ab) a target point cloud, c) template and target point cloud alignment via fast global registration (FGR), and d) template and target point cloud alignment via iterative closest point (ICP) and subsequent identification of scanning starting point (sphere marked with a "+") closest to the selected point (red sphere) in template.

[0042] Figure 9 is a flowchart showing an overview of an algorithm implemented for force management and optimising ultrasound probe orientation according to an embodiment of the invention.

[0043] Figures 10A and 10B are schematics showing how the orientation of the ultrasound probe is optimised through force feedback.

[0044] Figure 11 is flowchart giving an overview of the automation protocol according to an embodiment of the present invention.Detailed Description

[0045] Embodiments of the invention are discussed herein by reference to the drawings which are not to scale and are intended merely to assist with explanation of the invention.

[0046] The present invention introduces a novel approach to ultrasound scanning, which involves fusing images obtained using two image sensors having different imaging modalities. The fusion approach offers unique advantages since each imaging modality can capture different aspects and / or features of the subject and scene. The advantages include improved performance in varied conditions, enhanced detection and recognition, a reduction in ambiguity, and increased robustness.

[0047] The presence of a subject, being a human or animal subject, in a controlled environment, such as a clinical setting, can be identified by the body heat emitted by the subject. For example, humans have a core body temperature around 37°C (98.6°F), and therefore inherently radiate more heat than any inanimate objects, such as chairs and tables, which may be found in the surrounding environment. This thermal discrepancy enables a clear differentiation of the subject from the environment when viewed through a thermal image sensor. The same rationale applies to human and animal subjects, both of which radiate body heat.

[0048] A thermal sensor records radiation in the infrared spectrum, creating two- dimensional images that can display temperature variations across surfaces. Thermal sensors do not rely on external lighting and therefore are unaffected by ambient lighting. Whilst thermal images highlight temperature variations, they lack certain information critical for autonomous ultrasound imaging tasks such as ultrasound probe placement, pressure monitoring, anatomical navigation, and ensuring safety of the subject. This type of information may be obtained using a stereo depth camera. The inventors have identified that fusing thermal and depth image modalities provides comprehensive data that can enable autonomous ultrasound imaging.

[0049] Referring now to Figure 1A, there is shown an ultrasound imaging system according to an embodiment of the invention. The ultrasound imaging system 100 has a manipulator arm 102 to hold and move an ultrasound probe 104 along a body surface of thesubject (not shown, see Figure IB). The ultrasound imaging system 100 has at least two image sensors 106, with each image sensor using a different imaging modality to capture a different aspect of the subject. The image sensors 106 are mounted on an image sensor stand 108, to provide good visibility of the entire body of the subject for optimal imaging. Preferably, the image sensor stand can be adjusted in at least five degrees of freedom (DOFs).

[0050] The manipulator arm 102, may be part of a collaborative robot (or cobot) for manipulating the ultrasound probe 104. Cobots offer a combination of safety, adaptability, and user-friendliness, making them particularly well-suited to medical applications. A stand 110 is provided to support the cobot. The stand for supporting the cobot may or may not be height adjustable. An ultrasound examination table 112 is provided to support the subject or patient during the imaging procedure. An ultrasound charger 114 stand may be provided at one end of the cobot stand 110.

[0051] Referring now to Figure IB, there is shown the ultrasound imaging system 100 of Figure 1A in more detail. Subject 116 is lying prone on ultrasound examination table 112. Image sensors 106 mounted on image sensor stand 108 include a thermal sensor 118, the thermal sensor 118 adapted to capture a thermal image of the subject 116. Since a human or animal subject 116 will radiate body heat, they inherently radiate more heat than inanimate objects which may be present in the surrounding environment. Therefore, thermal sensor 118, can differentiate the subject 116 from the ultrasound examination table 112 upon which the subject 116 is prone.

[0052] Image sensors 106 further include a depth sensor 120, the depth sensor 120 configured to capture a depth image that provides a three-dimensional view of subject 116. The thermal and depth images together, provide control data for manoeuvring the manipulator arm 102 to position the ultrasound probe 104 for ultrasound imaging. A controller (see control box 120) is coupled to the manipulator arm 102 to cause the manipulator arm 102 to move the ultrasound probe 104 in accordance with the control data. The control data includes location information to identify an initial position on the body surface of the subject 116, for scanning the region of interest (ROI).

[0053] The initial position on the body surface of the subject 116 identified using the location information is a specific point on the surface of a subject 116, being located above the region of interest (ROI) for ultrasound scanning. Identification of this specific point, or initial position, enables the ultrasound probe 104 to be initially and autonomously positioned just above the initial position on the body surface of the subject 116, for example, at a small distance (say approximately 10 cm above) in order to prevent excessive pressure being applied or accidental injury to the subject 116. The ultrasound imaging system 100 subsequently lowers the ultrasound probe 104 to make controlled contact with the skin of the subject 116 using, for example, force and / or torque feedback mechanisms to regulate pressure and the like.

[0054] Also shown in Figure IB is a limit switch 122 incorporated to enhance safety in the event of a software malfunction by preventing the ultrasound probe 104 from reaching sensitive areas, such as the head of a subject 116. An emergency hand-held push button 124 is also included, allowing a human subject 116 to stop the ultrasound scanning procedure should they experience any discomfort during the imaging procedure.

[0055] Figure IB further includes a workstation 150. It will be appreciated that workstation 150 may be positioned in the same location, even the same room, as ultrasound imaging system 100, but alternatively could be remotely located. Workstation 150 may include a chair 152 and table 154 for the operator. A computer processor 156 for implementing functions to receive, manipulate and transmit data, including image data to be presented to the operator on a display 158. Input means such as a mouse 160 and / or keyboard 162 are further provided to enable the operator to input data such as the personal details identifying the subject 116 or similar, and / or instructions to cause the computer process or to execute functions.

[0056] Workstation 150 could be remotely located to the ultrasound imaging system 100. In particular, the ultrasound imaging system 100 lends itself to teleoperation by a remotely located operator. That is because in particular, the ultrasound imaging system generates control data including location information to identify an initial position for the ultrasound probe 104 on the body surface of the subject 116 for scanning the region of interest (ROI). Therefore, the ultrasound imaging system 100 using the proposed sensorfusion technique, can autonomously guide the ultrasound probe 104 towards the initial position or a start point on the subject 116 for subsequent teleoperation by a remotely located operator, or sonographer from that point onwards.

[0057] Any suitable ultrasound probe 104 may be employed. The ultrasound probe may be wired or enable wireless, real-time access to ultrasound imaging data captured directly from the probe. For example, the ultrasound probe 104 may include an Application Programming Interface (API) to enable such connectivity. A multipurpose Clarius C3 HD3 ultrasound probe is one suitable example. An advantage of using a wireless ultrasound probe 104 is that it may be autonomously detached from the manipulator arm post scanning, providing optimal flexibility and convenience in operation. However, it will be appreciated that a wired ultrasound probe could also be used.

[0058] The thermal sensor could be any suitable thermal camera that enables visualization of thermal radiation emitted by objects. For example, a forward looking infrared (FLIR) thermal camera, such as the compact FUR A50 smart sensor camera.

[0059] The depth sensor could be any suitable depth camera that capture depth data. For example, the depth camera could be a stereo camera or an RGB-D camera. An example of a suitable RGB-D camera is an Intel RealSense D415 RGB-D camera. The RealSense camera is compatible with various software development kits (SDKs) making them a suitable choice. It will be appreciated that the RGB-D camera has a module for capturing depth image data and also has an RGB imaging module for capturing RGB images.

[0060] Referring now to Figure 2, a communications network based on a Robot Operating System (ROS) is established to facilitate coordination between the image sensors 205, 210, manipulator arm 215 and ultrasound probe 220. The communications network 225 may employ programming languages Python and C++ to establish multiple nodes, facilitating real-time feedback acquisition from image sensors 205, 210 and enabling robot control. Two types of robot controllers may be utilized: read-only controllers 230 and commanding controllers 235. Read-only controllers 230 provide information on the cobot's current state, including its position, force-torque feedback, and status. Meanwhile, commanding controllers 235 direct the cobot's movements. To ensure safe operation, constraints are implemented on tool center point (TCP) speed and the range of joint movements.

[0061] The thermal image and the depth image captured by image sensors 205, 210, together are used to determine the control data that controls the cobot to capture ultrasound images 240, 245, 250, which are presented to an operator or sonographer via display 255. Depth can be sensed using a single three-dimensional depth image or a plurality of two-dimensional images.

[0062] To determine the control data, the thermal image and the depth images captured by image sensors 205, 210, are fused to form a fused depth-thermal composite image.Possible approaches for fusing a depth-thermal composite image include calibration-based fusion, feature-based fusion, and deep learning-based fusion. The calibration-based approach begins by estimating the intrinsic and extrinsic parameters of the image sensors, followed by preprocessing and subsequent rectification of the thermal and RGB images. Subsequently, depth data points are projected onto the thermal images, forming a 3D thermal point cloud. In feature-based fusion, features are extracted independently from both depth and thermal images. The extracted features are then combined or matched to form a comprehensive representation of the scene. Deep learning-based fusion uses neural networks to fuse depth and thermal data. The neural networks are trained to process both data types concurrently, to develop an optimal fusion strategy for applications, including object detection, segmentation, and classification.

[0063] Both the deep-learning and feature-based methods rely on the features of the scene and may encounter difficulties in scenes with unclear features or excessive noise, leading to potential errors and unpredictability in the fusion process. Furthermore, as each ultrasound imaging session presents a unique scene, the feature detection, description, and matching processes must be consistently performed for every image pair, which can be both time-consuming and computationally intensive.

[0064] Conversely, the calibration-based method relies on predetermined calibration parameters between the depth and thermal cameras. This approach is preferred since once the cameras are calibrated, there is no need for adjustments to be made in subsequent sessions, and the fusion process is not affected by variation in the scenes. Therefore, a novel calibration-based fusion approach was developed to enable autonomous robotic ultrasound imaging system. The proposed method prioritizes consistency and efficiency in image fusionacross different sessions. Some depth cameras capture colour (i.e., RGB) simultaneously with the depth images, producing an RGB-D image, as previously described.

[0065] The calibration approach developed by the inventors aligns the thermal image with the RGB image and then substitutes the RGB image with the thermal image. Thereafter the RGB-D camera's extrinsic (e.g., factory) calibration, which establishes a fixed relationship between the RGB and depth sensors within the RGB-D camera, is used to generate a fused depth-thermal composite image. This approach differs from known calibration-based alignment methods, which typically involve manually determining the extrinsic camera parameters.

[0066] Referring now to Figure 3, there is shown an overview of the depth-thermal sensor fusion approach. The process begins with the acquisition of a thermal image and a depth image. Preferably the thermal and depth images are captured simultaneously. The depth images may be acquired by an RGB-D camera, such that the depth image includes both RGB and depth image data, from which depth and RGB images can be extracted.

[0067] Accurate alignment of the RGB and thermal images is a precursor to the proposed sensor fusion approach. Preprocessing prepares the images for alignment and may involve rectification and resizing to match the thermal image and RGB image data. Preprocessing may also include noise filtering. A calibration procedure determines the intrinsic and extrinsic parameters of the two image sensors and rectifies the thermal and RBG images for subsequent alignment.

[0068] A checkerboard may be used to achieve calibration. In order to make the checkerboard detectable by both the thermal camera and the depth cameras simultaneously, a modified checkerboard is required, the checkerboard pattern being formed two materials having distinct colours and emissivities. For example, aluminium foil, having a silvery-white colour and an emissivity of around 0.04 may be used for one set of squares, while black paper having an emissivity of 0.94 may be used for the alternating squares.

[0069] During calibration, a plurality of sets of thermal and RGB images are captured simultaneously. Due to variation in the emissivities of the alternating materials in the checkerboard pattern, the black paper appears white, and the silvery-white aluminiumappears black in the thermal image, resulting in a difference in the origin of the thermal and RGB images. To address this issue, the colours of the thermal images are inverted to align them with the RGB images.

[0070] By way of example, using a FUR A50 camera to capture thermal images and an Intel RealSense D415 camera to capture the RGB-D images, the FUR A50 has a horizontal field of view (FOV) of 29° and a vertical FOV of 22°, with a resolution of 464 x 348, and the Intel RealSense D415 has a horizontal FOV of 65° and a vertical FOV of 40°, with a resolution of 1280 x 720. Given the differing FOVs, objects portrayed in the thermal images will appear more magnified than in the corresponding RGB images. To achieve overlap and vertical alignment between the RGB and thermal images, a scale factor, calculated from the ratio of the vertical FOVs of the two image sensors, is applied to both dimensions of the thermal images to resize them. To achieve dimension parity between the two sets of images, resized thermal images are padded with black margins, to bring their resolution to 1280 x 720.

[0071] The intrinsic parameters of the thermal and RGB images are then estimated. This estimation is required to undistort the images and ensure that they accurately reflect physical reality. This may be achieved, for example, using a MATLAB camera calibration app. Then, the extrinsic calibration parameters are estimated for rectification. This may be achieved, for example, using a MATLAB stereo camera calibration app. The MATLAB stereo camera calibrator app assumes a horizontal stereo camera setup, i.e., one camera beside the other. However, as shown in Figure IB, the two image sensors are spaced apart vertically, one above the other. In order to address this, both sets of images are rotated 90° to simulate a horizontal stereo setup. Once the calibration parameters are estimated and the images rectified, the rectified images are rotated 90° back to a vertical orientation.

[0072] Using the estimated calibration parameters, stereo rectification is performed to transform the images captured by the two image sensors such that their corresponding epipolar lines are vertically aligned. Post rectification, a point in one image will have its corresponding point located on the same vertical line in the corresponding image. This alignment is facilitated by the essential matrix (E), defined as:E — [ttdls * Rtd (1)where Rtdis the 3 x 3 rotation matrix, ttdis the translation vector estimated using the MATLAB stereo camera calibrator app that represents the thermal sensor's position relative to the depth sensor, and [ttd]sis the skew-symmetric matrix of ttd. By decomposing the matrix E, the rotation matrix Rdfor depth camera and Rtfor thermal camera can be calculated. These are the rectifying rotations that align the cameras to the new rectified coordinate system. After applying these rotations, the epipolar lines in the resulting images are parallel and vertically aligned. The projection matrices that remaps each point in the original images to its location in the rectified images can be represented as:Pd= KdRd[I | 0] (2)Pt=KtRt[Rtd| ttd] (3) where Pdand Ptare the projection matrices of the depth sensor and the thermal sensor, respectively. Similarly, Kdand Ktdenote the intrinsic matrices associated with the depth and thermal sensors, respectively and I represents the identity matrix.

[0073] Referring now to Figures 4A and 4B, even though stereo rectification will ensure that the epipolar lines are vertically aligned, a vertical difference, or disparity, will remain in the position of a specific point as seen between the two sensors. This disparity shown in Figure 4A is caused by the unique perspectives offered by each image sensor due to their vertical separation. In the case of image sensors that are vertically aligned, the disparity, d, can be estimated as:where f is the focal length of the camera, Bvis the baseline, which in this context refers to the vertical distance between the centers of the two cameras, and Z is the distance from the object point to the midpoint between the two image sensor camera centres.

[0074] A perfect alignment between the RGB and thermal images would require a disparity of zero, but achieving this in practice is impractical, since image sensors cannot occupy the exact same point in space. Therefore, two strategies are employed to minimize the disparity. Firstly, the image sensors are placed as closely together as possible, effectively reducing Bv. Secondly, the distance Z was reduced by positioning the cameras further awayfrom the region of interest (ROI) than absolutely necessary. As shown in Figure 4B, an estimation of the disparity between 0 to 5 m for this image sensor configuration, using Equation 4, demonstrates that there is a substantial reduction in disparity for distances greater than lm.

[0075] Following alignment of the RGB and thermal images, an algorithm is used to fuse the depth and thermal images. After capturing the RGB depth image as rs2::frameset, a temporal filter is applied to enhance depth image quality. This filter refines the depth values of a pixel by leveraging the temporal correlation between consecutive frames, resulting in a smoother and more stable depth output. Since there is no function to directly substitute an RGB image with a rectified thermal image, a simulation-based approach was developed utilizing the virtual RealSense device feature, rs2::software device, to allow for creation of custom depth and RGB data streams, that simulate the behaviour of a physical Intel RealSense D415 camera. The virtual RGB stream is populated with the rectified thermal image data, while the depth stream uses the actual depth data captured by the camera.These virtual streams are then synchronized within the simulation environment and captured as rs2::frameset with both depth information and the substituted thermal image.

[0076] Referring now to Figures 5A to 5D, the depth sensor's extrinsic parameters are used to fuse the depth and thermal images. Figure 5A represents the RGB image, and Figure 5B represents the thermal image. An algorithm is implemented to obtain the fused depththermal composite image shown in Figure 5C (and also in Figures 6A and 6B). The resulting image is presented as a mesh, shown in Figure 5D, which is a 3D representation of the captured scene, composed of interconnected vertices, edges, and faces.

[0077] Once the depth and thermal images have been fused, the intensity of the thermal data captured in the mesh face, is used to distinguish the subject lying on the ultrasound examination table from the background. For example, it is understood that in humans, a body temperature below 32°C indicates the life-threatening category of hypothermia. Therefore, any mesh face with a thermal intensity indicating a temperature below 32°C is cropped, as shown in Figure 5D. In the illustrated example, the exposed upper body is predominantly retained and some parts of the lower body. The extent of the lower bodycaptured is dependent on the clothing worn by the subject, which will influence the temperature readings by varying degrees.

[0078] Referring now to Figures 6A to 6D, mesh generation can introduce noise. Such noise may distort the true outer shape of the region of interest (ROI) that needs to be extracted and impact subsequent processing and analysis accordingly. The raw segmented depth-thermal mesh is shown in Figure 6A. Various filters have been applied to mitigate this issue, such as mean curvature flow, Laplacian smoothing, and Taubin filter. Mean curvature flow provides a geometric approach to mesh smoothing, but can be computationally intensive. Laplacian smoothing, while simple and effective at reducing minor noise, may lead to mesh shrinkage and potential ROI misidentification. The Taubin filter was designed to address the shrinkage issue of Laplacian smoothing's by alternating between contraction and expansion. It effectively reduces noise, maintains shape features, and eliminates the need for user-specific parameters. Considering its benefits, the Taubin filter was selected for mesh smoothing as is shown in Figure 6B.

[0079] To streamline the localization of the region of interest (ROI), the mesh is transformed into a point cloud as shown in Figure 6C and subsequently down sampled. This transformation simplifies the data to mere data points, omitting the complexities of edges and faces, and significantly reduces computational requirements. Voxel down sampling may be employed for this purpose. It involves dividing the 3D space into cubic segments or voxels, each represented by a single aggregated point, typically the centroid. This method ensures uniform representation for all regions of the point cloud, regardless of their original complexity or detail, thus maintaining the overall structural integrity and spatial relationships. Figure 6D shows the resultant reduced point cloud after down sampling.

[0080] Once the fused depth-thermal data has been pre-processed as described with reference to Figures 6A to 6D, the next step is to identify the region of interest from the data. In the illustrated examples, the region of interest in the lumbar region of a human subject, for the sake of simplicity. However, it will be understood that the method of autonomously moving an ultrasound probe to an initial position on the body surface of the subject for scanning a region of interest in a subject, is applicable to various other regions of interest.For example, the region of interest to be scanned could be the cardiac region, the abdominal region, a limb including an upper limb or a lower limb, amongst others.

[0081] Identifying the region of interest from the fused depth-thermal composite image involves use of a template matching algorithm. Template matching aligns a template point cloud to a target point cloud by comparing the geometric features of the two-point clouds. An advantage of the template matching algorithm is its simplicity, making it suitable for an autonomous system. The benefit compared to deep learning models is that it does not rely on training data. For example, where the region of interest is the lumbar region, the alignment process utilizes the entire back of a person as the template and identifies the lumbar region in subsequent steps. This approach provides a greater number of data points for alignment, leading to improved accuracy. Additionally, differentiating specific areas of the back is difficult in lower-resolution point clouds. For example, the upper and lower back regions sometimes appear similar in such data. By including the entire back, the risk of incorrectly identifying these areas is reduced.

[0082] Referring now to Figure 7, there is shown an overview of the template matching algorithm. The process is divided into a global and a local alignment phase. The global alignment provides a general alignment between the template and the target point cloud, which is the pre-processed and segmented fused depth-thermal point cloud. This initial alignment step is necessary because each point cloud is defined in a separate coordinate system, and the relationship between those coordinate systems is unknown.

[0083] Once a global alignment is achieved, the local alignment process is used to fine tune and ensure an optimal fit between the two-point clouds. Global alignment can be performed using the fast global registration (FGR) method, for example. FGR is an iterative algorithm that efficiently aligns two-point clouds by matching features between them. The output of the FGR process is a transformation matrix that includes a rotation matrix, RFGR, and a translation vector, IFGR, to align the template point cloud, Ptemp, with the target point cloud, Qtar. Alignment quality is evaluated using two metrics: normalized root mean square error (NRMSE) and fitness. Fitness measures the proportion of aligned points between Ptemp and Qtar, indicating the extent of their overall alignment. This fitness is calculated as:n(C)Fitness = (5) n(Ptemp) where C represents the set of inlier correspondences, n(C) is the total number of inlier correspondence pairs in C, and n(Ptemp) is the total number of points in Ptemp. A pair of points, one from Ptempand 'tsnearest match in Qtar, is considered inliers if the distance between them falls below a specified threshold. The threshold was selected based on alignment accuracy and inlier detection rates in test runs. The NRMSE measures the average error per correspondence and is calculated as:where p is a point in Ptemp foracorresponding matching point q in Qtar.

[0084] Consideration is given to variability in human anatomy, recognizing that differences in back sizes may lead to discrepancies during template matching. To address this issue, a method using adaptive scaling within the template matching algorithm was developed. This means that the template is adjusted in real-time to better fit the target data, leading to more accurate outcomes. At the beginning of the global alignment process, three versions of the template are created: an original template, a 10% larger version, and a 10% smaller version, and each is registered with the target point cloud. This facilitates determination as to whether the target point cloud is larger, smaller, or similar in size to the template. For this particular application, a fitness value of 0.9 (90%) was set as the threshold for alignment success. Amongst the three templates, the one having the highest fitness value exceeding 90% is selected to finalize the alignment, negating the need for further local alignment. Otherwise, the template having the smallest NRMSE from the enlarged and reduced templates, is selected for further local alignment. Switching the criterion to NRMSE ensures that the choice is made based on alignment quality.

[0085] For local alignment, an iterative closest point (ICP) algorithm is used. The ICP algorithm aims to find the best rigid transformation (rotation and translation) that aligns the template point cloud to the target point cloud. As an initial alignment for this process, the transformation estimated from fast global registration (FGR) is employed. ICP demonstrates robust performance in refining the alignment between two-point clouds to high precision,particularly when an approximate initial alignment is known or can be reliably estimated. Starting with a good initial alignment from global registration ensures that ICP converges faster and to a more accurate solution. Equation 5, which calculates fitness in the FGR process, is also used in the ICP registration. If the fitness between the template and target point cloud is below 90% after implementing ICP, the template point cloud undergoes a scaling either an increase or decrease of 10% based on the scaling used for the initial FGR analysis. This procedure is iterated until a fitness of at least 90% is achieved.

[0086] To position the ultrasound probe on the skin surface above the region of interest (ROI), i.e. the initial position determined by the control data, the relationship between the coordinate systems of the camera and robot must be determined. This is achieved through eye-to-hand calibration. Given that the system fuses depth and thermal images after aligning thermal and RGB images, the depth and RGB values at each pixel correspond to the same physical point in space. The RGB image is selected for calibration purposes due to its relative ease in target recognition compared to the depth image.

[0087] The calibration process utilises a ChArUco board as the calibration target. This board, printed in black and white, is mounted on a flat, sturdy surface and attached to the robot's flange. Using RoboDK software, the robot is programmed to move through a sequence of positions and orientations, ensuring that diverse viewpoints of the ChArUco board were captured by the camera from various angles and distances. Throughout this movement, both the robot's pose and images of the calibration target are simultaneously recorded. Subsequently, the captured images are processed with OpenCV library functions, which utilized the ArUco markers to accurately detect the corners of the checkerboard.

[0088] The detected corners and robot poses are used to estimate the transformation between the camera and robot coordinate systems. To initiate ultrasound scanning of the region of interest (ROI), a scanning starting point on the skin surface above the ROI must be located. For this purpose, the template point cloud used for the template matching algorithm is used. Referring now to Figure 8A, a sonographer, drawing from their knowledge of ultrasound imaging the lumbar region, selects a point on the skin surface of the subject, above the ROI in the template point cloud, marking it as the ideal scanning starting point. Once the template point cloud in Figure 8A has been aligned with the target point cloud inFigure 8B, the scanning starting point is located by aligning the template point could and the target point cloud via FGR as shown in Figure 8C, and then identifying the position in the target point cloud that is closest to the preselected point in the template point cloud, as shown in Figure 8D.

[0089] The optimal orientation of the ultrasound probe is to keep the probe perpendicular to the skin surface directly above the region of interest (ROI), this is defined as the initial orientation of the ultrasound probe on the body surface of the subject. Hence, after identifying the scanning starting point or initial position, the initial orientation of the ultrasound probe must be identified. A method based on local plane fitting is used to determine the initial orientation of the ultrasound probe. A plane in the fused depth-thermal point cloud is initially approximated by connecting the scanning starting point, P, and two closest points, A and B. A k-d tree is employed to simplify the computational process of identifying A and B. The ultrasound probe's optimal orientation vector is calculated as the outward normal vector to this plane. This normal unit vector, nnorm, is obtained as:> (A-P) x (B-P) nnorm -(A-P) X (B-P)I'

[0090] Given the fixed position of the image sensors relative to the ultrasound examination table, the orientation of the point cloud is known, with the positive z-direction extending outward from the skin surface. To ensure the correct outward-facing direction of nnorm, 'tsz-component is analyzed. If this component is negative, indicating an inward direction, the vector is reversed as:^norin— —^norin (8

[0091] After determining the normal vector, the rotation matrix that aligns this vector with the robot end-effector's vertical axis is computed. This matrix is used to determine the robots final pose in the form of a transformation matrix. The rotation should be around an axis that is perpendicular to both the given normal vector and the positive vertical axis. This axis, anorm, is computed using the cross product as:where z represents the positive vertical axis, given by the vector [0; 0; 1]T. To determine the exact angle of rotation between the given normal vector and the positive vertical axis, the dot product is utilized and the angle, 9, is obtained as:9 = arcos (z ■ nnorm) (10)

[0092] Thereafter, the Rodrigues' rotation formula is used to compute the rotation matrix, R, according to:where I denotes the identity matrix, and K is the skew-symmetric matrix derived from anorm = [anormx Hnormy Hnormz]^ according tO:

[0093] Considering patient safety, instead of autonomously moving the ultrasound probe directly onto the skin surface above the (region of interest) ROI, the manipulator arm initially moves the probe to a position approximately 10cm above this position on the surface of the subject in the normal direction. This precaution accounts for internal discrepancies or potential malfunctions in the robot. A Robot Operating System (ROS) based forward- cartesian-trajectory controller is employed to navigate the robot to this position.

[0094] After the ultrasound probe is positioned approximately 10cm above the initial position or scanning starting point, a ROS-based twist controller is used to establish contact between the ultrasound probe and the skin surface above the region of interest (ROI) and to reorient the ultrasound probe to the initial orientation as determined by the control data. During this process, it is prudent to manage the force exerted on the subject. A force range of 15 to 20N ensures an effective ultrasound scan without causing any discomfort to the subject. In initial tests, a force level of 15N proved appropriate for achieving good ultrasound imaging quality, thus it was selected as the reference force, Preference. As the ultrasound probe advances towards the scanning starting point along the robot end-effector's vertical axis, the linear velocity, vz, along this axis is regulated by the equation:where Fmeasured is the force measured by the force-torque sensor and Vmax is the maximum linear velocity set at 0.08 m / s. Hence, when Fmeasured is zero, vzis equal to Vmax. The forcetorque sensor was calibrated with the manipulator arm holding the ultrasound probe, ensuring that Fmeasured corresponds to the contact force. A deadband of IN was integrated. Thus, if the absolute value of vzis less than kz, vzis set to zero. This deadband ensures that minor deviations do not cause constant adjustments, which can cause unwanted vibrations

[0095] To further optimize the ultrasound probe's initial orientation, torque feedback is utilized. To execute the force management and this orientation refinement, a basic proportional force-torque control algorithm is implemented, as illustrated in Figure 9.

[0096] The orientation of the ultrasound probe is refined as is now described with reference to Figures 10A and 10B. Ensuring the ultrasound probe's orientation is perpendicular to the skin surface above the region of interest (ROI) requires that the torques around the x-axis, Tx, and the y-axis, Ty, are close to zero. Therefore, this value is designated as the reference torque for both Txand Ty. The governing equations for controlling the angular velocity to attain the reference torque, Treference, across both axes are:where coxand coydenote the angular velocities around the x and y axes, respectively. The coefficient kTis a proportional constant selected through iterative tuning. Meanwhile, tmeasured, x andTmeasured, y correspond to the measured torques about the x-axis and y- axis, respectively. A deadband of 0.3Nm was integrated into the system, mirroring the torque sensor's accuracy level.

[0097] Multithreading and callbacks are employed to concurrently read the force-torque sensor and implement all the twist control instructions. Furthermore, all the resulting twist commands are processed through a first order low-pass filter with a cutoff frequency of 210Hz to ensure commands are executed smoothly.

[0098] The entire process of lumbar region ultrasound imaging is managed by a script, for example a Python script, as is described by reference to Figure 11. First, the robot navigates to a predefined 'home' position. From this position, the manipulator arm picks up the ultrasound probe, subsequently returning to its home position. Simultaneously, both the depth camera and the thermal camera activate, capturing images of the subject lying prone on the ultrasound examination table. As previously described, control data determined by fusing the thermal and depth images is used to guide the ultrasound probe to the start position with optimal orientation. The proposed force-torque control algorithm can be modified to accommodate all basic ultrasound probe manipulation techniques. By default, the fanning technique is incorporated into the system as it can scan a broader area and provide a visualization of different segments of the lumbar region. While performing a fanning manipulation, a 10-second ultrasound video is recorded using the Clarius API. After recording the video, the ultrasound probe is maneuvered vertically upwards, ensuring it clears the subject prone on the bed. The manipulator arm then returns the ultrasound probe to its stand and moves back to the home position, signalling the conclusion of the automation process. To reduce the imaging session time, multithreading may be employed during image capture, processing, and automated pickup of the ultrasound probe.

[0099] The ultrasound imaging system of the present invention, utilises the subject's unique thermal pattern, combined with an innovative intensity-based algorithm and a depthbased template matching algorithm, to localise the region of interest. A force-torque algorithm is used to maintain the optimal force and optimize the ultrasound probe orientation. The system has demonstrated significant potential for application in clinical settings.

[0100] Where any or all of the terms "comprise", "comprises", "comprised" or "comprising" are used in this specification (including the claims), they are to be interpreted as specifying the presence of the stated features, integers, steps or components, but not precluding the presence of one or more other features, integers, steps or components.

[0101] While the invention has been described in conjunction with a limited number of embodiments, it will be appreciated by those skilled in the art that many alternatives, modifications and variations in light of the foregoing description are possible. Accordingly,the present invention is intended to embrace all such alternative, modifications and variations as may fall within the spirit and scope of the invention as disclosed.

Claims

The claims defining the invention are as follows:

1. An ultrasound imaging system for scanning a region of interest in a subject, the ultrasound imaging system comprising: a. an ultrasound probe; b. a manipulator arm to hold and move the ultrasound probe along a body surface of the subject; c. at least two image sensors, each image sensor using a different imaging modality to capture a different aspect of the subject for determining control data, wherein the at least two image sensors include a thermal sensor adapted to capture a thermal image to differentiate the body surface of the subject from the surrounding environment and a depth sensor adapted to capture one or more depth images to provide a three-dimensional view of the subject; and d. a controller coupled to the manipulator arm to cause the manipulator arm to move the ultrasound probe in accordance with the control data, wherein the control data includes location information to identify an initial position on the body surface of the subject for scanning the region of interest.

2. The ultrasound imaging system of claim 1, wherein the control data further includes orientation information to identify an initial orientation of the ultrasound probe on the body surface of the subject for scanning the region of interest.

3. The ultrasound imaging system of claim 1 or 2, wherein the control data further includes trajectory information to enable autonomous movement of the ultrasound probe from a home position to the initial position and initial orientation for scanning the region of interest.

4. The ultrasound imaging system of any one of claims 1 to 3, wherein the thermal image and the one or more depth images captured by the image sensors are fused to form a fused depth-thermal composite image.

5. The ultrasound imaging system of claim 4, wherein fused depth-thermal composite image is used to determine the control data.

6. The ultrasound imaging system of any one of claims 1 to 5, wherein the thermal image and the one or more depth images are captured substantially simultaneously.

7. The ultrasound imaging system of any one of claims 1 to 6, wherein the depth sensor comprises an RGB-D camera.

8. The ultrasound imaging system of any one of claims 1 to 7, wherein the manipulator arm is part of a collaborative robot.

9. A method of autonomously moving an ultrasound probe to an initial position on a body surface of a subject for scanning a region of interest in the subject, the method including the following steps: a. providing an ultrasound imaging system comprising an ultrasound probe, a manipulator arm to hold and move the ultrasound probe along the body surface of the subject, at least two image sensors, including a thermal sensor adapted to capture a thermal image to differentiate the body surface of the subject from the surrounding environment and a depth sensor adapted to capture one or more depth images to provide a three-dimensional view of the subject, and a controller coupled to the manipulator arm to cause the manipulator arm to move the ultrasound probe in accordance with a control data; b. acquiring at least a thermal image and one or more depth images; c. fusing the thermal image and the one or more depth images to determine the control data, wherein the control data includes location information to identify an initial position on the body surface of the subject for scanning the region of interest; and d. moving the ultrasound probe to the initial position on the body surface of the subject for scanning using the manipulator arm.

10. The method of claim 9, wherein the control data further includes orientation information to identify an initial orientation of the ultrasound probe on the body surface of the subject for scanning the region of interest.

11. The method according to claim 9 or 10, wherein the control data further includes trajectory information to enable autonomous movement of the ultrasound probe from a home position to the initial position and initial orientation for scanning of the region of interest.

12. The method according to any one of claims 9 to 11, wherein the step of moving the ultrasound probe to the initial position on the body surface of the subject further includes managing a force exerted on the surface of the body of the subject by the ultrasound probe by providing torque and force feedback to the manipulator arm.

13. The method according to any one of claims 9 to 12, wherein the step of fusing the thermal image and the one or more depth image provides a fused depth-thermal composite image.

14. The method according to any one of claims 9 to 13, wherein the step of acquiring images using the thermal sensor and the depth sensor occurs substantially simultaneously.

15. The method of any one of claims 9 to 14, wherein the depth sensor captures one or more RGB images using an RGB imaging module.

16. The method according to claim 15, wherein the step of fusing the thermal image with the one or more depth images to determine the control data further includes the following steps: a. estimating one or more parameters of the thermal sensor and the depth sensor; b. rectifying the thermal image and the depth image using the estimated parameters to achieve alignment between the thermal image and the RGB image; and c. substituting the RGB image with the rectified thermal image;d. projecting one or more depth data points extracted from the one or more depth images onto the rectified thermal image using one or more known calibration parameters of the depth sensor to form a fused depth-thermal composite image comprising a three-dimensional point cloud including location and thermal intensity information.

17. The method of claim 16, wherein the step of estimating one or more parameters of the thermal sensor and the depth sensor comprises estimating intrinsic and extrinsic parameters of the thermal sensor and the RGB imaging module of the depth sensor.

18. The method of claim 16 or 17, wherein the step of rectifying the thermal image and the RGB image is preceded by a preprocessing step.

19. The method of any one of claims 16 to 18, wherein the step of projecting one or more depth data points extracted from the depth image onto the rectified thermal image includes using the one or more known calibration parameters associated with the depth sensor to project depth image data acquired in the depth image onto the rectified thermal image.

20. A method of conducting an ultrasound scan of a region of interest in a subject, the method including the following steps: a. providing an ultrasound imaging system comprising an ultrasound probe, a manipulator arm to hold and move the ultrasound probe along a body surface of the subject, at least two image sensors, including a thermal sensor adapted to capture thermal image to differentiate the body surface of the subject from the surrounding environment and a depth sensor adapted to capture one or more depth images to provide a three-dimensional view of the subject, and a controller coupled to the manipulator arm to cause the manipulator arm to move the ultrasound probe in accordance with a control data; b. acquiring at least a thermal image and one or more depth images; c. fusing the thermal image and the one or more depth images to determine the control data, wherein the control data includes location information to identify an initial position on the body surface of the subject for scanning the region ofinterest and orientation information to identify an initial orientation of the ultrasound probe on the body surface of the subject; d. autonomously moving the ultrasound probe from a home position to the initial position and the initial orientation using the manipulator arm based on trajectory information provided by the control data; e. establishing contact between the ultrasound probe and the surface of the body of the subject using the manipulator arm; f. managing a force exerted on the surface of the body of the subject by the ultrasound probe by providing torque and force feedback to the manipulator arm; g. scanning the region of interest in the subject to provide a plurality of ultrasound images of the region of interest; and h. moving the ultrasound probe upwards using the manipulator arm and subsequently returning the ultrasound probe to the home position.

21. The method of claim 20, wherein the step of fusing the thermal image and the one or more depth images to determine the control data comprises fusing the thermal image with the one or more depth images to provide a fused depth-thermal composite image.

22. The method according to claim 21, wherein the step of acquiring the thermal image and the one or more depth images occurs substantially simultaneously.

23. The method according to any one of claims 20 to 22, wherein the step of managing a force exerted on the surface of the body of the subject by the ultrasound probe comprises use of a force-torque control algorithm.

24. The system of anyone of claims 1 to 8 or the method of any one of claims 9 to 23, wherein the subject is a human or animal subject.

25. The system or method of claim 24, wherein the region of interest is one or more of a lumbar region, a cardiac region, an abdominal region, or a limb including an upper limb and / or a lower limb.

Images