Methods and systems for analysis of ultrasound imaging using imaging and non-imaging features

Abstract

A method (100) for analyzing ultrasound imaging using an ultrasound analysis system (200), comprising: receiving (120), from an ultrasound imaging device, one or more ultrasound images obtained for a subject; receiving (130) non-imaging contextual information about: (1) the ultrasound imaging device or one or more parameters of the ultrasound images obtained for the subject; and / or (2) the subject; extracting (140) one or more imaging features from the received one or more ultrasound images; embedding (150) one or more non-imaging features from the received non-imaging contextual information; fusing (160) the extracted one or more imaging features and the embedded one or more non-imaging features into one or more fused features; analyzing (170) the one or more fused features to generate an ultrasound imaging analysis output; and providing (180), via a user interface of the ultrasound analysis system, the generated ultrasound imaging analysis output. The method further comprises utilizing (135), when certain non-imaging contextual information is not available: (1) one or more predetermined default values for unavailable non-imaging contextual information about the ultrasound imaging device and / or the one or more parameters of the ultrasound images obtained for the subject, and (2) one or more predetermined average values for unavailable non-imaging contextual information about the subject.

Description

[0001] 2025PF00013

[0002] 1

[0003] METHODS AND SYSTEMS FOR ANALYSIS OF ULTRASOUND IMAGING USING IMAGING AND NON-IMAGING FEATURES

[0004] FIELD OF THE INVENTION

[0005] The present disclosure is directed generally to methods and systems for analyzing ultrasound imaging using an ultrasound analysis system.

[0006] BACKGROUND OF THE INVENTION

[0007] Ultrasound technology has become a ubiquitous and indispensable tool in modem medicine. Its widespread adoption is due to its non-invasive nature, real-time imaging capabilities, and versatility in diagnosing and monitoring a variety of medical conditions. Ultrasound exams are routinely performed for a wide variety of analytical, diagnostic, and preventative reasons.

[0008] Traditional ultrasound image analysis algorithms rely predominantly on the visual information contained within ultrasound images themselves. These methods typically employ various computer vision and machine learning algorithms to identify relevant clinical image features. In contrast, human experts have the ability to combine visual information with a multitude of non-imaging contextual information to make clinical decisions. However, ultrasound image analysis algorithms fail to incorporate or consider this non-imaging contextual information when analyzing ultrasound images.

[0009] US 2025 / 111502 Al relates to contextual processing of ultrasound data. In particular, it discloses an ultrasound system that includes an ultrasound machine configured to generate, based on ultrasound data of a patient anatomy, an ultrasound image, and a processor system implemented to determine regional features, patient features, clinician features and / or ultrasound examination features, and generate, based on the ultrasound image and the determined features, a patient recommendation.

[0010] EP 4 501 245 Al relates to ultrasound imaging classification with size-based metrics, which enables a physician or other healthcare professional to quickly isolate, review, and confirm the most relevant fmdings / detections.

[0011] WO 2023 / 186584 Al discloses ultrasound-based structure localization using image and user inputs, while AU 2020230279 Al discloses an image analysis method and system for diagnosing and monitoring diseases or conditions in medical imaging in which a plurality of features extracted from both image and non-image data are combined using machine learning algorithms.

[0012] SUMMARY OF THE INVENTION

[0013] There is thus a continued need for methods and systems that consider both imaging information and non-imaging contextual information when analyzing ultrasound images.2025PF00013

[0014] 2

[0015] Various embodiments and implementations are directed to methods and systems for analyzing ultrasound imaging using an ultrasound analysis system. The system receives ultrasound images and non-imaging contextual information about the ultrasound imaging device or one or more parameters of the ultrasound images obtained for the subject, and / or about the subject. The system extracts imaging features from the ultrasound images, and embeds non-imaging features from the received non-imaging contextual information. The extracted imaging features and embedded non-imaging features are fused into one or more fused features, and the system analyzes the fused features to generate an ultrasound imaging analysis output. This output is provided to a user via a user interface of the ultrasound analysis system.

[0016] The invention is defined by the independent claims. Advantageous embodiments are provided in the dependent claims.

[0017] According to an aspect, a method for analyzing ultrasound imaging using an ultrasound analysis system is provided. The method includes: (i) receiving, from an ultrasound imaging device, one or more ultrasound images obtained for a subject; (ii) receiving non-imaging contextual information about: (1) the ultrasound imaging device or one or more parameters of the ultrasound images obtained for the subject; and / or (2) the subject; (iii) extracting one or more imaging features from the received one or more ultrasound images; (iv) embedding one or more non-imaging features from the received nonimaging contextual information; (v) fusing the extracted one or more imaging features and the embedded one or more non-imaging features into one or more fused features; (vi) analyzing the one or more fused features to generate an ultrasound imaging analysis output; and (vii) providing, via a user interface of the ultrasound analysis system, the generated ultrasound imaging analysis output.

[0018] The method further comprises utilizing, when certain non-imaging contextual information is not available: (1) one or more predetermined default values for unavailable non-imaging contextual information about the ultrasound imaging device and / or the one or more parameters of the ultrasound images obtained for the subject, and (2) one or more predetermined average values for unavailable nonimaging contextual information about the subject.

[0019] In other words, if certain non-imaging contextual information is missing or unavailable, utilizing instead predetermined default values for unavailable non-imaging contextual information about the ultrasound imaging device and / or the one or more parameters of the obtained ultrasound images and predetermined average values for unavailable non-imaging contextual information about the subject.

[0020] The step of utilizing one or more predetermined default / average values for unavailable non-imaging contextual information is preferably performed before the step of embedding one or more non-imaging features from the received non-imaging contextual information.

[0021] The present invention recognizes the problem that complete non-imaging contextual information may not always be available and, hence, cannot be provided to, an analysis algorithm to make clinical decisions and / or predictions. The method of the present invention solves this problem by using predetermined default or average values if certain non-imaging contextual information is unavailable or2025PF00013

[0022] 3

[0023] only partially available. This ensures that the image analysis can continue to function effectively and generate meaningful diagnostic outputs even in absence of certain contextual data, such as specific subject data.

[0024] In this way, the present invention advantageously enhances the reliability and usability of ultrasound image analysis even in scenarios where complete non-imaging contextual data cannot be provided, improving robustness and ensuring continuity of operation (i.e., reducing the risk of system or algorithm failure) in data-limited environments.

[0025] According to an embodiment, the non-imaging contextual information is received from a clinician via the user interface of the ultrasound analysis system. In this manner, expert clinical knowledge can be advantageously integrated directly into the analysis process. This ensures that the system can incorporate nuanced, real-time information that may not be readily available from automated sources, thereby improving the accuracy and relevance of the analysis.

[0026] According to an embodiment, non-imaging contextual information about the ultrasound imaging device or one or more parameters of the ultrasound images obtained for the subject comprises one or more of transducer type, ultrasound zone or location, and information from another analysis algorithm. This may enable the system to account for device-specific or procedural variations, which enhances the robustness of the analysis by tailoring it to the imaging conditions, and to adapt accordingly, hence, reducing potential errors caused by variability in imaging setups. According to an embodiment, the information from another analysis algorithm comprises a segmentation output. Incorporating information from another analysis algorithm, such as segmentation outputs, offers the technical advantage of leveraging complementary data to refine the analysis. This multi-source integration improves the system's ability to detect and interpret complex patterns, leading to more precise and reliable diagnostic outputs.

[0027] According to an embodiment, non-imaging contextual information about the subject comprises one or more of subject demographics, subject medical history, subject medical condition(s), and subject test result(s). The use of this non-imaging contextual information about the subject makes it possible to personalize the analysis to the individual subject. This contextualization may advantageously enhance diagnostic accuracy by considering subject-specific factors that may influence the interpretation of ultrasound images.

[0028] According to an embodiment, the one or more imaging features are extracted from the received one or more ultrasound images using a trained machine learning algorithm. The use of a trained machine learning algorithm for extracting imaging features enables automating the feature extraction process with high precision and consistency. This may advantageously reduce reliance on manual interpretation, minimize human error, and enable the system to process large volumes of data efficiently.

[0029] According to an embodiment, analyzing the one or more fused features to generate an ultrasound imaging analysis output comprises analyzing the one or more fused features with a trained machine learning algorithm. In this manner, imaging and non-imaging data can be combined in a sophisticated manner to generate more comprehensive and accurate diagnostic outputs. This fusion2025PF00013

[0030] 4

[0031] approach mimics the decision-making process of clinical experts, improving the system's diagnostic capabilities.

[0032] According to some embodiments, embedding the one or more non-imaging features from the received non-imaging contextual information comprises transforming the one or more non-imaging features into respective one or more non-imaging feature vectors. In this way, the non-imaging features can be brought into a format that is particularly amenable for integration with the format of the imaging features.

[0033] In one of such embodiments, fusing the extracted one or more imaging features and the embedded one or more non-imaging features into one or more fused features comprises: spatially repeating the / each non-imaging feature vector to match a spatial dimensions of a feature map generated from the / each extracted imaging feature; and concatenating the spatially repeated one or more nonimaging feature vectors with the one or more feature maps for the extracted one or more imaging features along a feature dimension to generate a plurality of fused features.

[0034] In another of such embodiments, fusing the extracted one or more imaging features and the embedded one or more non-imaging features into one or more fused features comprises: spatially repeating the / each non-imaging feature vector to match a spatial dimensions of a feature map generated from the / each extracted imaging feature; and combining, using a convolutional layer, the spatially repeated one or more non-imaging feature vectors and the one or more feature maps for the extracted one or more imaging features to generate a plurality of fused features.

[0035] According to an embodiment, the received non-imaging contextual information about the ultrasound imaging device or one or more parameters of the ultrasound images obtained for the subject comprises one or more predetermined default values. That is, according to this embodiment, the received non-imaging contextual information about the ultrasound imaging device or one or more parameters of the ultrasound images obtained for the subject is incomplete and has been augmented to comprise one or more predetermined default values.

[0036] According to an embodiment, the received non-imaging contextual information about the subject comprises one or more predetermined average values. That is, according to this embodiment, the received non-imaging contextual information about the subject is incomplete and has been augmented to comprise one or more predetermined average values.

[0037] According to another aspect of the present invention, an ultrasound analysis system is provided. The system includes: an ultrasound imaging device configured to obtain one or more ultrasound images for a subject; a user interface configured to receive input from a clinician; and a processor configured to: (i) receive, from the ultrasound imaging device, the one or more ultrasound images obtained for a subject; (ii) receive non-imaging contextual information about: (1) the ultrasound imaging device or one or more parameters of the ultrasound images obtained for the subject; and / or (2) the subject; (iii) extract one or more imaging features from the received one or more ultrasound images; (iv) embed one or more non-imaging features from the received non-imaging contextual information; (v) fuse the2025PF00013

[0038] 5

[0039] extracted one or more imaging features and the embedded one or more non-imaging features into one or more fused features; (vi) analyze the one or more fused features to generate an ultrasound imaging analysis output; and (vii) provide the generated ultrasound imaging analysis output via the user interface.

[0040] The processor is further configured to utilize, when certain non-imaging contextual information is not available: (1) one or more predetermined default values for unavailable non-imaging contextual information about the ultrasound imaging device and / or the one or more parameters of the ultrasound images obtained for the subject, and (2) one or more predetermined average values for unavailable non-imaging contextual information about the subject.

[0041] According to an embodiment, the processor is configured to receive the non-imaging contextual information from a clinician via the user interface of the ultrasound analysis system.

[0042] According to yet another aspect of the present invention, a non-transitory machine-readable medium is provided. The medium stores instructions which, when executed by a processor, cause the processor to perform the method according to any of the embodiments described above.

[0043] It should be appreciated that all combinations of the foregoing concepts and additional concepts discussed in greater detail below (provided such concepts are not mutually inconsistent) are contemplated as being part of the inventive subject matter disclosed herein. In particular, all combinations of claimed subject matter appearing at the end of this disclosure are contemplated as being part of the inventive subject matter disclosed herein. It should also be appreciated that terminology explicitly employed herein that also may appear in any disclosure incorporated by reference should be accorded a meaning most consistent with the particular concepts disclosed herein.

[0044] These and other aspects of the various embodiments will be apparent from and elucidated with reference to the embodiment(s) described hereinafter.

[0045] BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In the drawings, like reference characters generally refer to the same parts throughout the different views. The figures showing features and ways of implementing various embodiments and are not to be construed as being limiting to other possible embodiments falling within the scope of the attached claims. Also, the drawings are not necessarily to scale, emphasis instead generally being placed upon illustrating the principles of the various embodiments.

[0047] Fig. 1 is a flowchart of a method for analyzing ultrasound imaging, in accordance with an embodiment.

[0048] Fig. 2 is a schematic representation of an ultrasound analysis system, in accordance with an embodiment.

[0049] Fig. 3 is a flowchart of a method for analyzing ultrasound imaging, in accordance with an embodiment.

[0050] Fig. 4 is a flowchart of a method for analyzing ultrasound imaging, in accordance with an embodiment.2025PF00013

[0051] 6

[0052] Fig. 5 is a flowchart of a method for analyzing ultrasound imaging, in accordance with an embodiment.

[0053] Fig. 6 is a flowchart of a method for analyzing ultrasound imaging, in accordance with an embodiment.

[0054] Fig. 7 is a flowchart of a method for analyzing ultrasound imaging, in accordance with an embodiment.

[0055] Fig. 8 is a flowchart of a method for training a machine learning algorithm, in accordance with an embodiment.

[0056] DETAILED DESCRIPTION OF EMBODIMENTS

[0057] The present disclosure describes various embodiments of a system and method configured to analyze ultrasound imaging. More generally, Applicant has recognized and appreciated that it would be beneficial to analyzing ultrasound imaging using both imaging information and non-imaging contextual information. Accordingly, an ultrasound analysis system receives ultrasound images and nonimaging contextual information about the ultrasound imaging device or one or more parameters of the ultrasound images obtained for the subject, and / or about the subject. If certain non-imaging contextual information is not available, the system utilizes predetermined default values for unavailable non-imaging contextual information about the ultrasound imaging device and / or the one or more parameters of the ultrasound images obtained for the subject, and predetermined average values for unavailable nonimaging contextual information about the subject. The system extracts imaging features from the ultrasound images, and embeds non-imaging features from the received non-imaging contextual information. The extracted imaging features and embedded non-imaging features are fused into one or more fused features, and the system analyzes the fused features to generate an ultrasound imaging analysis output. This output is provided to a user via a user interface of the ultrasound analysis system.

[0058] The methods and systems described or otherwise envisioned herein comprise ultrasound image analysis that fuses image features with non-imaging information to improve diagnostic accuracy. By integrating non-imaging data sources, the system can account for factors that influence image appearance and interpretation. For example, knowing the type of ultrasound probe used, or the anatomical zone / location being scanned, can help adjust for specific imaging characteristics. Incorporating this additional non-imaging context reduces dependence on image quality, allows the system to align more closely with the decision-making of clinical experts, and mitigates risk in challenging clinical scenarios where diagnostic misclassifications would otherwise occur. Thus, the methods and systems described or otherwise envisioned herein not only improve the ultrasound image analysis system, they improve patient safety and care.

[0059] The embodiments and implementations disclosed or otherwise envisioned herein can be utilized with any system or process that may utilize or benefit from improved ultrasound imaging analysis. The embodiments and implementations disclosed or otherwise envisioned herein can be utilized2025PF00013

[0060] 7

[0061] with any system that generates or analyzes ultrasound imaging, including but not limited to Philips® Lumify® or Compact® (manufactured by Koninklijke Philips, N.V.), among other products and imaging modalities. The embodiments and implementations disclosed or otherwise envisioned herein can be utilized for point-of-care implementations including lung, FAST, and cardiac applications, among many others. However, the disclosure is not limited to these devices, systems, or implementations, and thus the disclosure and embodiments disclosed herein can encompass any system that may utilize or benefit from the improved ultrasound imaging analysis.

[0062] Referring to Fig. 1, in one embodiment, is a flowchart of a method for analyzing ultrasound imaging using an ultrasound analysis system. The methods described in connection with the figures are provided as examples only, and shall be understood not to limit the scope of the disclosure. The ultrasound analysis system can be any of the systems described or otherwise envisioned herein. The ultrasound analysis system can be a single system or multiple different systems.

[0063] At step 110 of the method, an ultrasound analysis system 200 is provided. Referring to an embodiment of an ultrasound analysis system 200 as depicted in Fig. 2, for example, the system comprises one or more of a processor 220, memory 230, user interface 240, communications interface 250, and storage 260, interconnected via one or more system buses 212. It will be understood that Fig. 2 constitutes, in some respects, an abstraction and that the actual organization of the components of the system 200 may be different and more complex than illustrated. Additionally, ultrasound analysis system 200 can be any of the systems described or otherwise envisioned herein. Other elements and components of the ultrasound analysis system 200 are disclosed and / or envisioned elsewhere herein.

[0064] According to an embodiment, the ultrasound analysis system 200 comprises or is in direct or indirect communication with an imaging modality 270. The imaging modality can be, for example, an ultrasound system such as EPIQ® system, Philips Lumify®, and Philips Affinity®, among many others. The images obtained using the imaging modality may be obtained from a clinical provider or other individual. For example, the user of the imaging modality may be an experienced sonographer, a firsttime user of ultrasound, or any experience level in between.

[0065] According to an embodiment, the ultrasound analysis system 200 comprises or is in direct or indirect communication with an image database 280. The image database may be any image database, and may comprise images or videos or reports or other data obtained using any imaging modality, such as imaging modality 270. The image database 280 may be local to the ultrasound system, and may optionally be a component of the system. The image database 280 may alternatively be remote to the ultrasound system, and thus is in direct or indirection communication with the ultrasound system.

[0066] Referring to Fig. 3, in one embodiment, is a high-level overview of a method 300 for analyzing ultrasound imaging using an ultrasound analysis system 200. At 310, the system receives or obtains ultrasound imaging and extracts one or more features from the received ultrasound imaging at 320 to generate imaging features 340. At 320, the system receives or obtains non-imaging contextual information and embeds one or more features from the non-imaging contextual information at 330 to2025PF00013

[0067] 8

[0068] generate the non-imaging features at 350. The imaging features 340 and non-imaging features 350 are fused to generate fused features 360.

[0069] At step 120 of method 100 in Fig. 1, ultrasound imaging of the patient or subject is obtained using the imaging modality 270 of the ultrasound system 200. The patient or subject can be any subject that could potentially benefit from ultrasound. According to one embodiment, the ultrasound imaging may be obtained of any region of the subject’s body, such as the abdomen, chest, head, or limbs among other regions. The ultrasound imaging may be obtained using any method, system, or technique for obtaining ultrasound imaging. The ultrasound imaging comprises a plurality of frames per second. Once obtained, the ultrasound imaging may be utilized immediately, and / or it may be temporarily or permanently stored in local and / or remote memory for future use, including in image database 280.

[0070] At step 130 of the method, the ultrasound analysis system 200 obtains or receives nonimaging contextual information. The non-imaging contextual information is about or regarding one or more of: (1) the ultrasound imaging device; (2) one or more parameters of the ultrasound images obtained for the subject; and (3) the subject.

[0071] For example, non-imaging contextual information about the ultrasound imaging device can be information about the imaging device itself, a transducer type, and / or other information about the imaging modality, among other information. The non-imaging contextual information about one or more parameters of the ultrasound images obtained for the subject can be information about the ultrasound zone or location that was visualized during the ultrasound imaging, and / or information such as an analysis or prediction or estimate from another algorithm or system, such as detection / segmentation outputs for other ultrasound features, among other information. The non-imaging contextual information about the subject can be clinical information such as patient demographics (e.g., age, gender, size), medical history, medical condition(s), and / or test result(s), among other information.

[0072] The system can receive the non-imaging contextual information in a variety of ways. For example, the system can automatically obtain contextual information from a component of the ultrasound system, or remotely from a database of contextual information. For contextual information about a subject, the system can automatically obtain the contextual information from an electronic health or medical record database, from information entered into the system about the subject, or from any other source. According to another embodiment, the system receives or obtains contextual information from a user interface. For example, a user or clinician may enter information about the subject, the ultrasound system, or the ultrasound examination into a user interface of the system, or into a user interface of a separate system that is in direct or indirect communication with the ultrasound analysis system. This information may be entered as part of the ultrasound examination, or as part of the ultrasound analysis method described or otherwise envisioned herein. For example, a clinician may be guided through a series of one or more questions that request the non-imaging contextual information.

[0073] Referring to Fig. 4, for example, is a high-level overview of a method 400 for analyzing ultrasound imaging using an ultrasound analysis system 200, in which the non-imaging contextual2025PF00013

[0074] 9

[0075] information is a user-selected scanning zone provided as input to a feature fusion step. In this example, the user can select a specific zone during an ultrasound scan using a user interface of the ultrasound analysis system. At 410, the system receives or obtains ultrasound imaging and extracts one or more features from the received ultrasound imaging at 420 to generate imaging features 440. At 420, the system receives or obtains non-imaging contextual information, including identification of a user-selected scanning zone (i.e., R1 in the white box), and embeds one or more features from the non-imaging contextual information at 430 to generate the non-imaging features at 450. The imaging features 440 and non-imaging features 450 are fused to generate fused features 460. In this embodiment, the selected zone (i.e., Rl) indicates a right-anterior scan on the thoracic (chest) area. This is encoded and provided as input to the feature fusion step, which computes a prediction by combining the non-image contextual information (i.e., the identification of the Rl zone) with the ultrasound image.

[0076] Returning to method 100 in Fig. 1, at step 140 of the method the ultrasound analysis system extracts one or more image features from the received ultrasound imaging. This can be performed in a variety of different ways, including methods known in the art. According to one embodiment, the image feature extraction can be performed using a convolutional neural network (CNN), or any other trained machine learning algorithm. According to an embodiment, the output of the image feature extraction is a feature vector that can be used in downstream steps of the method. As just one non-limiting example, the output of the image feature is an N*width*height feature vector that encapsulates visual information from the image, is relevant to the downstream classification task. However, other image features are possible. According to an embodiment, traditional image descriptors may be included alongside the extracted features.

[0077] Once generated, the one or more image features may be utilized immediately, and / or they may be temporarily or permanently stored in local and / or remote memory for future use.

[0078] At step 150 of the method, the system embeds one or more non-imaging features from the received non-imaging contextual information. This can be performed in a variety of different ways, including methods known in the art. According to one embodiment, non-imaging data are transformed into a format amenable to integration with imaging features. Non-imaging data can be summarized into the following categories, among others: (1) categorical data, such as zone information and probe type, which are converted to one-hot encoded vectors; and (2) continuous variables, such as patient age and lab test results, which are normalized (e.g., by their mean and standard deviations) and embedded as numerical vectors. According to an embodiment, these non-imaging data are combined to form an M-dimensional feature vector capturing the non-imaging contextual information.

[0079] At step 160 of the method, the system fuses the extracted one or more imaging features and the embedded one or more non-imaging features into one or more fused features. This can be performed in a wide variety of ways. Although several methods for fusing the imaging features and embedded non-imaging features into fused features are described below, it will be understood that these are not the only methods for fusion. According to an embodiment, therefore, the imaging and non-2025PF00013

[0080] 10

[0081] imaging features are combined before a secondary classification model (such as a second CNN) for refinement and downstream classification is utilized to analyze the fused features.

[0082] Referring to Fig. 5, in one embodiment, is a high-level overview of a first fusion method 500 in which non-imaging features are spatially repeated and concatenated with imaging features.

[0083] According to an embodiment, non-imaging feature vectors are spatially repeated to match the spatial dimensions of feature maps generated from the imaging features. The spatially repeated non-imaging features are concatenated with the imaging feature maps along the feature dimension, thereby creating a combined feature map that integrates information from both data sources. Thus, referring to Fig. 5, the imaging features 510 (N channel*width*height) are concatenated with spatially repeated features 530 (M channel*width*height) generated from non-imaging features 520, to generate fused features 540 (N+M channel * width* height) .

[0084] Referring to Fig. 6, in one embodiment, is a high-level overview of a second fusion method 600 in which imaging features and non-imaging features are combined in a convolutional layer. Non-imaging feature vectors are spatially repeated to match the spatial dimensions of the imaging feature maps. A 3x3 convolutional kernel from a subsequent convolutional layer is then applied to both the imaging and non-imaging features, resulting in a new layer with fused features. This new feature map has dimensions of N*width*height, integrating data from both imaging and non-imaging data. Thus, referring to Fig. 6, the imaging features 610 (N channel*width*height) are concatenated in a convolutional layer with spatially repeated features 630 (M channel*width*height) generated from non-imaging features 620, to generate fused features 640 (N channel*width*height).

[0085] Referring to Fig. 7, in one embodiment, is a high-level overview of a third fusion method 600 in which non-imaging features used as input conditions for models trained with imaging features alone. According to this embodiment, a machine learning model is developed only using imaging features. However, as a post-processing step, the model accepts additional inputs from non-imaging features, enabling it to adjust the final predictions. For instance, specific zones in ultrasound video might provide higher confidence thresholds for predicting the presence of certain pathologies. Thus, referring to Fig. 7, the imaging features 710 (N channel*width*height) are used by atrained machine learning model 730, which also accepts non-imaging features 720 to generate one or more outputs or predictions 740 regarding the ultrasound imaging.

[0086] According to an embodiment, the extracted one or more imaging features and the embedded one or more non-imaging features may not be available. There are many reasons why these features may not be available or accessible as expected. There may be an issue with communication within the system, a clinician may not have an answer to a question about the procedure, subject, or system, or there may be a software or hardware issue with the system, among other causes.

[0087] Returning to method 100 in Fig. 1, at step 135 of the method the system utilizes, when certain non-imaging contextual information is not available, one or more predetermined default values for unavailable non-imaging contextual information about the ultrasound imaging device and / or the one or2025PF00013

[0088] 11

[0089] more parameters of the ultrasound images obtained for the subject, and one or more predetermined average values for unavailable non-imaging contextual information about the subject. That is, at step 135 of the method, the system verifies whether the received non-imaging contextual information is complete. If it is incomplete, then the system utilizes predetermined default values or predetermined average values for the missing or unavailable data.

[0090] Accordingly, the ultrasound analysis system can still function when, for example, nonimaging contextual information is unavailable or only partially available. In this instance, where the nonimaging contextual information is not available, a default value is utilized. For example, ultrasound zones can be coded as all-zero embeddings, and patient age can be replaced with the mean age of all subjects in the dataset. During inference, users have the option to input non-imaging features into the system. In a similar manner, the model can make predictions using default values if non-imaging features are not available.

[0091] At step 170 of the method, the system analyzes the one or more fused features to generate an ultrasound imaging analysis output. According to an embodiment, the methods and systems described or otherwise envisioned herein can be utilized to generate fused features (fused from imaging features and non-imaging contextual information features as described above) that can be used in a wide variety of different ways. For example, the ultrasound system or another system may comprise an analytical or diagnostic algorithm which utilizes the one or more fused features for analysis or diagnosis, such as a clinical prediction, estimate, or other information. There are many different algorithms or software packages that can utilize extracted or generated features to analyze ultrasound imaging.

[0092] For example, an ultrasound analysis algorithm of the system may utilize the one or more fused features for lesion detection, such as for identifying or highlighting abnormal areas, such as tumors or cysts, within ultrasound images. An ultrasound analysis algorithm of the system may utilize the one or more fused features for organ segmentation, such as for automatically outlining and segmenting organs like the liver, kidneys, or heart to assist in detailed analysis. An ultrasound analysis algorithm of the system may utilize the one or more fused features for disease classification, such as classifying the type of disease or condition present, including distinguishing between benign and malignant tumors. An ultrasound analysis algorithm of the system may utilize the one or more fused features for quality assessment, such as evaluating the quality of the ultrasound images to ensure they are suitable for diagnosis, and flagging images that may be problematic. An ultrasound analysis algorithm of the system may utilize the one or more fused features for measurement and quantification, such as providing precise measurements of structures within the body, including the size of a fetus during pregnancy or the thickness of arterial walls. Many other analyses and outputs are possible.

[0093] Once generated, the ultrasound imaging analysis output may be utilized immediately, and / or it may be temporarily or permanently stored in local and / or remote memory for future use.

[0094] At step 180 of the method, the generated ultrasound imaging analysis output is provided to a clinician or user via a user interface of the ultrasound analysis system. The provided output can be2025PF00013

[0095] 12

[0096] any of the information as described or otherwise envisioned herein, including but not limited to the ultrasound imaging analysis output resulting from analysis of the fused features. The system may provide the information to a user via any mechanism, including but not limited to a visual display, an audible notification, a page, or any other method of notification. The information may be communicated by wired and / or wireless communication to another device. For example, the system may communicate the information to a mobile phone, computer, laptop, wearable device, and / or any other device configured to allow display and / or other communication of the information.

[0097] According to an embodiment, the methods and systems described or otherwise envisioned herein can be utilized to generate fused features (fused from imaging features and non-imaging contextual information features as described above) that can be used in a wide variety of different ways. Methods and systems utilizing fused features can be identified in a variety of different ways. For example, according to one embodiment, a system may allow or request a user to input non-imaging data such as lung zones, patient demographics, medical history, and lab results. The presence of these prompts in the user interface indicates the system utilizes non-imaging data for predictions or other analysis of ultrasound imaging. As another example, according to one embodiment, ultrasound images acquired under different contexts appear visually similar. If an ultrasound analysis system is able to provide diagnostic results in these scenarios, the system is combining imaging and non-imaging data for the analysis.

[0098] According to an embodiment, the ultrasound analysis system 200 comprises one or more trained machine learning algorithms. For example, the algorithm analyzing the one or more fused features to generate an ultrasound imaging analysis output can comprise analyzing the one or more fused features with a trained machine learning algorithm.

[0099] Referring to Fig. 8, in one embodiment, is a flowchart of a method 800 for training a machine learning algorithm of the ultrasound analysis system 200 to utilize fusion features to analyze ultrasound imaging. The method may be performed by the ultrasound system, or may be performed by another system such as a specialized machine learning model training system. According to one embodiment, the machine learning algorithm is trained using a supervised or unsupervised training process.

[0100] At step 810 of the method, the training system receives training data which will be used to train the model. The training data can be any data sufficient to train the model to utilize the described input data to generate the described output. For example, the training data may comprise ultrasound imaging and fusion features associated with the ultrasound imaging, generated pursuant to the methods and systems described or otherwise envisioned herein. This training data, which could be utilized in a supervised or unsupervised manner, can comprise imaging for 100s or 1000s of patients and / or procedures, and can be updated with new imaging. The training data will also comprise other information, such as non-imaging contextual information and features. This training data may be curated by an expert such as a clinician, or it may be obtained and utilized without curation. The training data may be received2025PF00013

[0101] 13

[0102] from any source. For example, the training data may be received from an electronic medical record database or system, or any other component of the system or a training system. According to an embodiment, system 200 comprises or is in direct or indirect communication with a database which comprises some or all of the training data set.

[0103] According to an embodiment, the training system may comprise a data pre-processor or similar component or algorithm configured to process the received training data. For example, the data pre-processor analyzes the training data to remove noise, bias, errors, and other potential issues. The data pre-processor may also analyze the input data to remove low quality data. Many other forms of data preprocessing or data point identification and / or extraction are possible.

[0104] At step 820 of the method, the training system trains the machine learning algorithm, using the training data, to generate the respective output data from input data. The algorithm is trained using any method fortraining such a model. The trained algorithm is a unique model based on the training data used to train the model. Following training, the system comprises a trained machine learning algorithm. Thus, following training, the algorithm is a specialized model configured to receive the specialized input (described above) and generate the very specific output (also described above).

[0105] At step 830 of the method, the trained algorithm is stored for future use. According to an embodiment, the trained algorithm may be stored in local or remote storage.

[0106] Referring again to Fig. 2 is a schematic representation of an ultrasound imaging system 200. System 200 may be any of the systems described or otherwise envisioned herein, and may comprise any of the components described or otherwise envisioned herein. It will be understood that Fig. 2 constitutes, in some respects, an abstraction and that the actual organization of the components of the system 200 may be different and more complex than illustrated.

[0107] According to an embodiment, system 200 comprises a processor 220 capable of executing instructions stored in memory 230 or storage 260 or otherwise processing data to, for example, perform one or more steps of the method. Processor 220 may be formed of one or multiple modules. Processor 220 may take any suitable form, including but not limited to a microprocessor, microcontroller, multiple microcontrollers, circuitry, field programmable gate array (FPGA), application-specific integrated circuit (ASIC), a single processor, or plural processors.

[0108] Memory 230 can take any suitable form, including a non-volatile memory and / or RAM. The memory 230 may include various memories such as, for example LI, L2, or L3 cache or system memory. As such, the memory 230 may include static random access memory (SRAM), dynamic RAM (DRAM), flash memory, read only memory (ROM), or other similar memory devices. The memory can store, among other things, an operating system. The RAM is used by the processor for the temporary storage of data. According to an embodiment, an operating system may contain code which, when executed by the processor, controls operation of one or more components of system 200. It will be apparent that, in embodiments where the processor implements one or more of the functions described2025PF00013

[0109] 14

[0110] herein in hardware, the software described as corresponding to such functionality in other embodiments may be omitted.

[0111] User interface 240 may include one or more devices for enabling communication with a user. The user interface can be any device or system that allows information to be conveyed and / or received, and may include a display, a mouse, and / or a keyboard for receiving user commands. In some embodiments, user interface 240 may include a command line interface or graphical user interface that may be presented to a remote terminal via communication interface 250. The user interface may be located with one or more other components of the system, or may located remote from the system and in communication via a wired and / or wireless communications network.

[0112] Communication interface 250 may include one or more devices for enabling communication with other hardware devices. For example, communication interface 250 may include a network interface card (NIC) configured to communicate according to the Ethernet protocol.

[0113] Additionally, communication interface 250 may implement a TCP / IP stack for communication according to the TCP / IP protocols. Various alternative or additional hardware or configurations for communication interface 250 will be apparent.

[0114] Storage 260 may include one or more machine-readable storage media such as read-only memory (ROM), random-access memory (RAM), magnetic disk storage media, optical storage media, flash-memory devices, or similar storage media. In various embodiments, storage 260 may store instructions for execution by processor 220 or data upon which processor 220 may operate. For example, storage 260 may store an operating system 261 for controlling various operations of system 200.

[0115] It will be apparent that various information described as stored in storage 260 may be additionally or alternatively stored in memory 230. In this respect, memory 230 may also be considered to constitute a storage device and storage 260 may be considered a memory. Various other arrangements will be apparent. Further, memory 230 and storage 260 may both be considered to be non-transitory machine-readable media. As used herein, the term non-transitory will be understood to exclude transitory signals but to include all forms of storage, including both volatile and non-volatile memories.

[0116] While system 200 is shown as including one of each described component, the various components may be duplicated in various embodiments. For example, processor 220 may include multiple microprocessors that are configured to independently execute the methods described herein or are configured to perform steps or subroutines of the methods described herein such that the multiple processors cooperate to achieve the functionality described herein. Further, where one or more components of system 200 is implemented in a cloud computing system, the various hardware components may belong to separate physical systems. For example, processor 220 may include a first processor in a first server and a second processor in a second server. Many other variations and configurations are possible.

[0117] According to an embodiment, system 200 comprises or is in direct or indirect communication with an imaging modality 270. The imaging modality can be any modality sufficient to2025PF00013

[0118] 15

[0119] obtain imagery utilized by the ultrasound system 200 to obtain ultrasound imagery. The imaging modality can be, for example, an ultrasound system such as EPIQ® system, Philips Lumify®, and Philips Affinity®, among many others.

[0120] According to an embodiment, system 200 comprises or is in direct or indirect communication with an image database 280. The image database may be any image database, and may comprise images or videos or reports or other data obtained using any imaging modality, such as imaging modality 270. The image database 280 may be local to the ultrasound system, and may optionally be a component of the system. The image database 280 may alternatively be remote to the ultrasound system.

[0121] According to an embodiment, storage 260 of system 200 may store one or more algorithms, modules, and / or instructions to carry out one or more functions or steps of the methods described or otherwise envisioned herein. For example, storage 260 may comprise, among other instructions or data, extraction and embedding instructions 262, feature fusion instructions 263, a trained machine learning algorithm 264, and / or reporting instructions 265.

[0122] According to an embodiment, extraction and embedding instructions 262 direct the system to extract one or more image features from the received ultrasound imaging. This can be performed in a variety of different ways, including methods known in the art. According to one embodiment, the image feature extraction can be performed using a convolutional neural network (CNN), or any other trained machine learning algorithm. According to an embodiment, the output of the image feature extraction is a feature vector that can be used in downstream steps of the method. As just one nonlimiting example, the output of the image feature is an N*width*height feature vector that encapsulates visual information from the image, is relevant to the downstream classification task. However, other image features are possible. According to an embodiment, traditional image descriptors may be included alongside the extracted features.

[0123] The extraction and embedding instructions 262 also direct the system to embed one or more non-imaging features from the received non-imaging contextual information. This can be performed in a variety of different ways, including methods known in the art. According to one embodiment, nonimaging data are transformed into a format amenable to integration with imaging features. Non-imaging data can be summarized into the following categories, among others: (1) categorical data, such as zone information and probe type, which are converted to one-hot encoded vectors; and (2) continuous variables, such as patient age and lab test results, which are normalized (e.g., by their mean and standard deviations) and embedded as numerical vectors. According to an embodiment, these non-imaging data are combined to form an M-dimensional feature vector capturing the non-imaging contextual information.

[0124] According to an embodiment, feature fusion instructions 263 direct the system to fuse the extracted one or more imaging features and the embedded one or more non-imaging features into one or more fused features. This can be performed in a wide variety of ways. Although several methods for fusing the imaging features and embedded non-imaging features into fused features are described below, it will be understood that these are not the only methods for fusion. According to an embodiment,2025PF00013

[0125] 16

[0126] therefore, the imaging and non-imaging features are combined before a secondary classification model (such as a second CNN) for refinement and downstream classification is utilized to analyze the fused features.

[0127] According to an embodiment, the trained machine learning algorithm 264 is an algorithm of the ultrasound analysis system 200 that utilizes fusion features to analyze ultrasound imaging. The method may be performed by the ultrasound system, or may be performed by another system such as a specialized machine learning model training system.

[0128] According to an embodiment, reporting instructions 265 direct the system to provide the output of the system to a user, such as a clinician, via a user interface. The provided output can be any of the information as described or otherwise envisioned herein, including but not limited to the ultrasound imaging analysis output resulting from analysis of the fused features. The system may provide the information to a user via any mechanism, including but not limited to a visual display, an audible notification, a page, or any other method of notification. The information may be communicated by wired and / or wireless communication to another device. For example, the system may communicate the information to a mobile phone, computer, laptop, wearable device, and / or any other device configured to allow display and / or other communication of the information.

[0129] According to an embodiment, the ultrasound analysis system 200 is configured to process many thousands or millions of datapoints in input data used to train one or more of the machine learning algorithms. For example, generating a functional and skilled trained algorithm from a corpus of training data requires processing of millions of datapoints from input data and generated features. This can require millions or billions of calculations to generate a novel trained algorithm from those millions of datapoints and millions or billions of calculations. As a result, each trained algorithm is novel and distinct based on the input data and parameters of the model, and thus improves the functioning of the system. Generating a functional and skilled trained algorithm comprises a process with a volume of calculation and analysis that a human brain cannot accomplish in a lifetime, or multiple lifetimes.

[0130] Additionally, the ultrasound imaging analysis system 200 is configured to process many thousands or millions of datapoints in the ultrasound imaging in order to generate the output information, including but not limited to the ultrasound imaging analysis output resulting from analysis of the fused features. This comprises a volume of calculation and analysis that a human brain cannot accomplish in a lifetime, or multiple lifetimes. Further, the real-time nature of the claimed method and system also comprises an analysis that a human brain cannot accomplish with or without pen / pencil or paper.

[0131] All definitions, as defined and used herein, should be understood to control over dictionary definitions, definitions in documents incorporated by reference, and / or ordinary meanings of the defined terms.

[0132] The indefinite articles “a” and “an,” as used herein in the specification and in the claims, unless clearly indicated to the contrary, should be understood to mean “at least one.”2025PF00013

[0133] 17

[0134] The phrase “and / or,” as used herein in the specification and in the claims, should be understood to mean “either or both” of the elements so conjoined, i.e., elements that are conjunctively present in some cases and disjunctively present in other cases. Multiple elements listed with “and / or” should be construed in the same fashion, i.e., “one or more” of the elements so conjoined. Other elements may optionally be present other than the elements specifically identified by the “and / or” clause, whether related or unrelated to those elements specifically identified.

[0135] As used herein in the specification and in the claims, “or” should be understood to have the same meaning as “and / or” as defined above. For example, when separating items in a list, “or” or “and / or” shall be interpreted as being inclusive, i.e., the inclusion of at least one, but also including more than one, of a number or list of elements, and, optionally, additional unlisted items. Only terms clearly indicated to the contrary, such as “only one of’ or “exactly one of,” or, when used in the claims, “consisting of,” will refer to the inclusion of exactly one element of a number or list of elements. In general, the term “or” as used herein shall only be interpreted as indicating exclusive alternatives (i.e. “one or the other but not both”) when preceded by terms of exclusivity, such as “either,” “one of,” “only one of,” or “exactly one of.”

[0136] As used herein in the specification and in the claims, the phrase “at least one,” in reference to a list of one or more elements, should be understood to mean at least one element selected from any one or more of the elements in the list of elements, but not necessarily including at least one of each and every element specifically listed within the list of elements and not excluding any combinations of elements in the list of elements. This definition also allows that elements may optionally be present other than the elements specifically identified within the list of elements to which the phrase “at least one” refers, whether related or unrelated to those elements specifically identified.

[0137] It should also be understood that, unless clearly indicated to the contrary, in any methods claimed herein that include more than one step or act, the order of the steps or acts of the method is not necessarily limited to the order in which the steps or acts of the method are recited.

[0138] In the claims, as well as in the specification above, all transitional phrases such as “comprising,” “including,” “carrying,” “having,” “containing,” “involving,” “holding,” “composed of,” and the like are to be understood to be open-ended, i.e., to mean including but not limited to. Only the transitional phrases “consisting of’ and “consisting essentially of’ shall be closed or semi-closed transitional phrases, respectively.

[0139] While several inventive embodiments have been described and illustrated herein, those of ordinary skill in the art will readily envision a variety of other means and / or structures for performing the function and / or obtaining the results and / or one or more of the advantages described herein, and each of such variations and / or modifications is deemed to be within the scope of the inventive embodiments described herein. More generally, those skilled in the art will readily appreciate that all parameters, dimensions, materials, and configurations described herein are meant to be exemplary and that the actual parameters, dimensions, materials, and / or configurations will depend upon the specific application or2025PF00013

[0140] 18

[0141] applications for which the inventive teachings is / are used. Those skilled in the art will recognize, or be able to ascertain using no more than routine experimentation, many equivalents to the specific inventive embodiments described herein. It is, therefore, to be understood that the foregoing embodiments are presented by way of example only and that, within the scope of the appended claims and equivalents thereto, inventive embodiments may be practiced otherwise than as specifically described and claimed. Inventive embodiments of the present disclosure are directed to each individual feature, system, article, material, kit, and / or method described herein. In addition, any combination of two or more such features, systems, articles, materials, kits, and / or methods, if such features, systems, articles, materials, kits, and / or methods are not mutually inconsistent, is included within the inventive scope of the present disclosure.

Claims

2025PF0001319CLAIMS:

1. A method (100) for analyzing ultrasound imaging using an ultrasound analysis system (200), comprising:receiving (120), from an ultrasound imaging device, one or more ultrasound images (310, 410) obtained for a subject;receiving (130) non-imaging contextual information (320, 420) about: (1) the ultrasound imaging device or one or more parameters of the ultrasound images obtained for the subject; and / or (2) the subject;extracting (140) one or more imaging features (340, 440, 510, 610, 710) from the received one or more ultrasound images;embedding (150) one or more non-imaging features (350, 450, 520, 620, 720) from the received non-imaging contextual information;fusing (160) the extracted one or more imaging features and the embedded one or more non-imaging features into one or more fused features (360, 460, 540, 640);analyzing (170) the one or more fused features to generate an ultrasound imaging analysis output; andproviding (180), via a user interface (240) of the ultrasound analysis system, the generated ultrasound imaging analysis output,wherein the method further comprises utilizing (135), when certain non-imaging contextual information is not available:- one or more predetermined default values for unavailable non-imaging contextual information about the ultrasound imaging device and / or the one or more parameters of the ultrasound images obtained for the subject, and- one or more predetermined average values for unavailable non-imaging contextual information about the subject.

2. The method of claim 1, wherein the non-imaging contextual information (320, 420) is received from a clinician via the user interface (240) of the ultrasound analysis system.

3. The method of claim 1 or 2, wherein non-imaging contextual information about the ultrasound imaging device or one or more parameters of the ultrasound images obtained for the subject comprises one or more of transducer type, ultrasound zone or location, and information from another analysis algorithm.2025PF00013204. The method of claim 3, wherein the information from another analysis algorithm comprises a segmentation output.

5. The method of any of claims 1 to 4, wherein non-imaging contextual information about the subject comprises one or more of subject demographics, subject medical history, subject medical condition(s), and subject test result(s).

6. The method of any of claims 1 to 5, wherein the one or more imaging features (340, 440, 510, 610, 710) are extracted from the received one or more ultrasound images (310, 410) using a trained machine learning algorithm.

7. The method of any of claims 1 to 6, wherein analyzing (170) the one or more fused features to generate an ultrasound imaging analysis output comprises analyzing the one or more fused features with a trained machine learning algorithm.

8. The method of any of claims 1 to 7, wherein embedding (150) the one or more nonimaging features from the received non-imaging contextual information comprises transforming the one or more non-imaging features (520, 620) into respective one or more non-imaging feature vectors.

9. The method of claims 8, wherein fusing (160) the extracted one or more imaging features and the embedded one or more non-imaging features into one or more fused features comprises:spatially repeating the / each non-imaging feature vector to match a spatial dimensions of a feature map generated from the / each extracted imaging feature ( 10); andconcatenating the spatially repeated one or more non-imaging feature vectors (530) with the one or more feature maps for the extracted one or more imaging features ( 10) along a feature dimension to generate a plurality of fused features (540).

10. The method of claims 8, wherein fusing (160) the extracted one or more imaging features and the embedded one or more non-imaging features into one or more fused features comprises:spatially repeating the / each non-imaging feature vector to match a spatial dimensions of a feature map generated from the / each extracted imaging feature (610); andcombining, using a convolutional layer, the spatially repeated one or more non-imaging feature vectors (630) and the one or more feature maps for the extracted one or more imaging features (610) to generate a plurality of fused features (640).

11. An ultrasound analysis system (200), comprising:2025PF0001321an ultrasound imaging device (270) configured to obtain one or more ultrasound images (310, 410) for a subject;a user interface (240) configured to receive input from a clinician;a processor (220) configured to: (i) receive, from the ultrasound imaging device, the one or more ultrasound images obtained for a subject; (ii) receive non-imaging contextual information (320, 420) about: (1) the ultrasound imaging device or one or more parameters of the ultrasound images obtained for the subject; and / or (2) the subject; (iii) extract one or more imaging features (340, 440, 510, 610, 710) from the received one or more ultrasound images; (iv) embed one or more non-imaging features (350, 450, 520, 620, 720) from the received non-imaging contextual information; (v) fuse the extracted one or more imaging features and the embedded one or more non-imaging features into one or more fused features (360, 460, 540, 640); (vi) analyze the one or more fused features to generate an ultrasound imaging analysis output; and (vii) provide the generated ultrasound imaging analysis output via the user interface,wherein the processor (220) is further configured to utilize, when certain non-imaging contextual information is not available:- one or more predetermined default values for unavailable non-imaging contextual information about the ultrasound imaging device and / or the one or more parameters of the ultrasound images obtained for the subject, and- one or more predetermined average values for unavailable non-imaging contextual information about the subject.

12. The system of claim 11, wherein the processor (220) is configured to receive the nonimaging contextual information (320, 420) from a clinician via the user interface (240) of the ultrasound analysis system.

13. The system of claim 11 or 12, wherein non-imaging contextual information about the ultrasound imaging device or one or more parameters of the ultrasound images obtained for the subject comprises one or more of transducer type, ultrasound zone or location, and information from another analysis algorithm.

14. The system of any of claims 11 to 13, wherein non-imaging contextual information about the subject comprises one or more of subject demographics, subject medical history, subject medical condition(s), and subject test result(s).

15. A non-transitory machine-readable medium (260) storing instructions which, when executed by a processor (220), cause the processor to perform the method of any of claims 1 to 10.

Images