Continuous machine learning retraining using massive medical data

Abstract

Techniques of repeatedly re-training a medical machine learning system using a repeatedly updated stream of training data are presented wherein the techniques include: acquiring at least one additional training data sample; updating a training data buffer based on the at least one additional training data sample, and discarding at least one training data sample from the training data buffer corresponding to the number of the at least one additional training data sample, and retraining the medical machine learning system with at least one mini-batch from the training data buffer.

Description

Attorney Docket No. P18428-02 / 0184.0329-PCTCONTINUOUS MACHINE LEARNING RETRAINING USING MASSIVE MEDICAL DATARelated Application

[0001] This application claims the benefit of U. S. Provisional Patent Application No. 63 / 720,403 entitled “CONTINUOUS MACHINE LEARNING RETRAINING USING MASSIVE MEDICAL DATA,” filed November 14, 2024.Field

[0002] This disclosure relates generally to machine learning, e.g., in the medical field.Background

[0003] Increasing amounts of massive medical data, which may be used to train medical machine learning systems, are becoming publicly available. Such massive medical data may include an increasing number of scans (e.g., from imaging technologies such as X-ray, CT, MEI, PET, etc.), may originate from varying sources (e.g., diverse medical centers and research facilities), and may incorporate additional classes (e.g., as image segmentation and / or image classification become more sophisticated). With the increasing availability of massive medical data, prevalent machine learning training paradigms — where machine learning systems are trained with multiple passes over fixed, finite datasets — face significant challenges.

[0004] For example, existing machine learning paradigms are largely designed for fixed, finite datasets. However, training machine learning systems all at once onall available massive medical data is impractical, if not outright intractable, as new scans / sources / classes continuously arrive.

[0005] As another example of challenges that arise from the increasing availability of massive medical data, training machine learning systems continuously on new scans / sources / classes can lead to catastrophic forgetting, where a machine learning system forgets old data as it learns new data, and vice versa. Such catastrophic forgetting may occur when an existing machine learning system is retrained using new scans, sources, and / or classes.

[0006] Attempts to mitigate catastrophic forgetting have been conducted in the domain of natural images. Regularization-based methods constrain model flexibility, typically through techniques applied to weights and gradients, or via knowledge distillation focused on output logits and intermediate features. However, these techniques fail to ensure optimal performance on challenging tasks, particularly without stored exemplars. Expandable models attempt to address this by either incrementally growing with new tasks or maintaining a common backbone model adaptable through small task-specific adaptation blocks. However, these approaches lead to inflated model sizes and bloated network architectures. In contrast, rehearsal techniques involve storing and replaying a small subset of training samples, embedded features, or generators from prior tasks. Nonetheless, these techniques are not suitable for continuously streaming sources of new data.Summary

[0007] According to various embodiments, a method of repeatedly re-training a medical machine learning system using a repeatedly updated stream of training data is presented. The medical machine learning system may have been previously trainedusing previous training data samples. The method may include: acquiring at least one additional training data sample from at least one online streaming training data source; updating a training data buffer based on the at least one additional training data sample, where the updating includes: adding a number of the at least one additional training data sample to the training data buffer, determining respective significance values for respective training data samples in the training data buffer, and discarding at least one training data sample from the training data buffer corresponding to the number of the at least one additional training data sample, where the discarding includes discarding based on a pairwise similarity comparison of training data samples in the training data buffer and based on the respective significance values; and retraining the medical machine learning system with at least one mini-batch from the training data buffer.

[0008] Various optional features of the above method embodiments include the following. The method may include: providing a patient data sample to the medical machine learning system, from which a patient output is obtained; and providing an indication of the patient output. The method may include repeating the acquiring, the updating, and the retraining on an ongoing basis, whereby the medical machine learning system is continuously retrained based on the repeatedly updated stream of training data. The medical machine learning system may include a machine learning image segmentation system. A respective significance value may include a respective weighted sum of class entropy values for segmentation classes in a respective image. The respective weighted sum of class entropy values may include a respective penalized weighted sum of binary cross entropy values. A respective significance value for a respective training data sample may be based on a misclassification of the respective training data sample by the medical machine learning system. Theacquiring may include acquiring the at least one additional training data sample from a plurality of online streaming training data sources. A respective training data sample may include a three-dimensional image. A respective pairwise similarity comparison may include a cosine similarity of a respective pair of feature vectors of a respective pair of training data samples.

[0009] According to various embodiments, a system for repeatedly re-training a medical machine learning system using a repeatedly updated stream of training data is presented. The medical machine learning system may have been previously trained using previous training data samples. The system for repeatedly re-training the medical machine learning system includes: a non-transitory computer readable medium including instructions; and at least one electronic processor that executes the instructions to perform operations including: acquiring at least one additional training data sample from at least one online streaming training data source; updating a training data buffer based on the at least one additional training data sample, where the updating includes: adding a number of the at least one additional training data sample to the training data buffer, determining respective significance values for respective training data samples in the training data buffer, and discarding at least one training data sample from the training data buffer corresponding to the number of the at least one additional training data sample, where the discarding includes discarding based on a pairwise similarity comparison of training data samples in the training data buffer and based on the respective significance values; and retraining the medical machine learning system with at least one mini-batch from the training data buffer.

[0010] Various optional features of the above system embodiments include the following. The operations may further include: providing a patient data sample to the medical machine learning system, from which a patient output is obtained; andproviding an indication of the patient output. The operations may further include repeating the acquiring, the updating, and the retraining on an ongoing basis, whereby the medical machine learning system is continuously retrained based on the repeatedly updated stream of training data. The medical machine learning system may include a machine learning image segmentation system. A respective significance value may include a respective weighted sum of class entropy values for segmentation classes in a respective image. The respective weighted sum of class entropy values may include a respective penalized weighted sum of binary cross entropy values. A respective significance value for a respective training data sample may be based on a misclassification of the respective training data sample by the medical machine learning system. The acquiring may include acquiring the at least one additional training data sample from a plurality of online streaming training data sources. A respective training data sample may include a three-dimensional image. A respective pairwise similarity comparison may include a cosine similarity of a respective pair of feature vectors of a respective pair of training data samples.

[0011] Combinations, (including multiple dependent combinations) of the above-described elements and those within the specification have been contemplated by the inventors and may be made, except where otherwise indicated or where contradictory.Brief Description of the Drawings

[0012] Various features of the examples can be more fully appreciated, as the same become better understood with reference to the following detailed description of the examples when considered in connection with the accompanying figures, in which:

[0013] Fig. 1 is a schematic diagram illustrating linear memory, according to various embodiments;

[0014] Fig. 2 is a schematic diagram illustrating dynamic memory, according to various embodiments;

[0015] Fig. 3 is a schematic diagram illustrating selective memory, according to various embodiments;

[0016] Fig. 4 presents charts illustrating catastrophic forgetting during repeated retraining among machine learning systems that incorporate linear memory, dynamic memory, and selective memory, according to example implementations;

[0017] Fig. 5 presents charts illustrating the diversity of retained samples during repeated retraining, according to example implementations;

[0018] Fig. 6 is a flowchart for a method of updating a training data buffer based on additional training data samples from a repeatedly updated stream of training data, according to various embodiments; and

[0019] Fig. 7 is a flowchart for a method of repeatedly re-training a medical machine learning system using a repeatedly updated stream of training data, according to various embodiments.Description of the Examples

[0020] Reference will now be made in detail to example implementations, illustrated in the accompanying drawings. Wherever convenient, the same reference numbers will be used throughout the drawings to refer to the same or like parts. In the following description, reference is made to the accompanying drawings that form a part thereof, and in which is shown by way of illustration specific exemplary examples in which the invention may be practiced. These examples are described in sufficientdetail to enable those skilled in the art to practice the invention and it is to be understood that other examples may be utilized and that changes may be made without departing from the scope of the invention. The following description is, therefore, merely exemplary.

[0021] Some embodiments provide for repeatedly re-training a medical machine learning system using a repeatedly updated stream of medical training data. Some embodiments retrain an existing trained machine learning system using only the most significant samples from an updated stream of medical training data, based on the uniqueness of the data samples and the machine learning system’s prediction uncertainty. Compared with existing training paradigms, some embodiments not only improve data efficiency by allowing for training on continual data streams, thereby solving the problem of existing system requiring training on a fixed set of data, but also solve the problem of catastrophic forgetting by selectively training machine learning systems on significant data samples that might otherwise be forgotten.

[0022] Some embodiments incorporate one or more of the following four advantages over existing techniques. First, some embodiments handle the presence of incomplete annotations, which is a frequently-occurring issue that arises because public medical datasets are often only partially labeled due to the high cost of annotating medical images at the voxel level. Existing techniques typically require fully-labeled training data. Second, some embodiments are able to learn from new scans / sources without forgetting old ones. Existing techniques typically rely on epochs — a complete, shuffled pass through the entire dataset — which are impractical for continuously updated data sources. Existing techniques often require a machine learning system to process the entire dataset multiple times. Third, some embodiments incorporate training data with new classes without repeated retraining,which is advantageous because many medical images that are not fully labeled may be sometime in the future. Existing machine learning techniques typically ignore the possibility of new classes. Fourth, some embodiments allow for retraining machine learning systems using simple, manageable resources, which is advantageous because clinical environments usually lack large data storage and computational capacities. Existing machine learning technique require extensive memory for data selection and class extension.

[0023] Embodiments may provide one or more of the aforementioned advantages, thus solving the aforementioned problems of prior art techniques, as follows.

[0024] Some embodiments provide the first and third advantages though the use of text encoding to segment classes based on their text descriptions, allowing embodiments to handle new classes and tasks within partially labeled datasets by simply modifying the text descriptions. Some embodiments provide the second advantage by storing recent samples in a training data buffer as new data becomes available, allowing for training continuously from new data without revisiting old data and avoiding the need for complete passes through the entire dataset (as shown and described herein in reference to Fig. 1). Some embodiments provide the first and fourth advantages by storing particularly beneficial training data, by selecting samples based on their uniqueness (e.g., based on a pairwise similarity comparison), which prevents catastrophic forgetting by preserving samples from different time periods and sources (as shown and described in reference to Fig. 2). Some embodiments provide the fourth advantage by deliberately identifying and using samples with high uncertainty (e.g., according to an entropy function), which improves performance ofthe trained machine learning system without the need for extra computational resources (as shown and described in reference to Fig. 3).

[0025] The inventors have tested a non-limiting example embodiment a large-scale single-site private dataset and a sequential-site dataset composed of 16 public datasets and report the results of the testing herein. On the single-site dataset, the testing demonstrates that the example embodiment achieved comparable performance to the prevalent training paradigm that trains the model over multiple passes (82.2% vs. 82.6%). Moreover, the example embodiment enhances data efficiency by allowing for training continuously from new data without revisiting old data (as described in reference to Table 1). On the sequential-site dataset with varying sources of data, a 6% performance increase was observed by effectively selecting unique data compared to directly storing recent samples (as shown and described in reference to Table 2). The example embodiment also successfully mitigated catastrophic forgetting by enhancing data diversity and preserving previous knowledge (as shown and described in reference to Figs. 4 and 5). Furthermore, by deliberately learning samples with high uncertainty, the example embodiment boosted performance by 9%, which outperforms the prevalent training paradigm.

[0026] Fig. 1 is a schematic diagram illustrating linear memory 100, according to various embodiments. As shown in Fig. 1, a medical machine learning system 104 hosted on a computer is trained on training data samples. The medical machine learning system may include any, or a combination, of: a neural network, a transformer, a random forest, a decision tree, and / or a support vector machine.

[0027] Each streaming training data source 112, 114, 116, 118 may include any, or a combination, of a website, an API, a web service, and / or a database. Non-limiting example curators of streaming training data sources 112, 114, 116, 118 may includehospitals, research institutions, government archives, etc. According to various embodiments, the training data samples available from the streaming training data sources 112, 114, 116, 118 may include medical data, such as, by way of non-limiting examples: two-dimensional medical images (e.g., X-ray images, ultrasound images), three-dimensional medical images (e.g., CT volumes, MRI volumes, PET volumes, etc.), clinical medical data (e.g., test results, demographics data, etc.), and / or continuous medical data (e.g., spectroscopic data, time-based data, etc.). According to various embodiments, the training data samples may be unlabeled, may be partially labeled, or may be fully labeled. The contents of the online streaming training data sources 112, 114, 116, 118 may be repeatedly updated on a continuing basis, e.g., continuously. The online streaming training data sources 112, 114, 116, 118 may be characterized as streaming in that additional samples may be added to them after previously-added samples have already been used, according to an embodiment.

[0028] According to some embodiments, the medical machine learning system 104 is trained using a training buffer 102 that stores select training data samples. The number of training data samples stored in the training buffer 102 may be fixed, according to some embodiments, and may be relatively small compared to the number of training data samples available from the online streaming training data sources 112, 114, 116, 118. The training data samples stored in the training data buffer 102 are retrieved from the repeatedly updated stream of training data provided by the plurality of online streaming training data sources 112, 114, 116, 118 though a network 106 (e.g., the internet). The process by which training data samples are selected and stored in the training data buffer 102 is presented in detail throughout this disclosure and may include one or more of: linear memory (e.g., as shown and described presently in reference to Fig. 1), dynamic memory (as shown and described inreference to Fig. 2), and / or selective memory (as shown and described in reference to Fig. 3).

[0029] To illustrate the difference between linear memory 100 and the prior art, a description of a typical known training paradigm follows. In general, the prevalent machine learning training paradigm typically relies on fixed-size datasets, characterized by their finiteness, immutability, and full availability at all times. Leveraging these attributes, according to the prevalent training paradigm, samples within such datasets can be indexed, shuffled, and accessed in full throughout the training process. Further, the prevalent training paradigm often needs iterating over the datasets multiple times. However, online learning based on a repeatedly updated stream of training data, which embraces a data stream potentially of infinite length, necessitates a departure from the prevalent training paradigm. Within this context, data retrieval from a streaming source at any given moment t yields the current sample xt, with future samples inaccessible and past samples retrievable only if stored upon acquisition.

[0030] Some embodiments solve certain problems of the prevalent machine learning training paradigm through the use of linear memory 100. Linear memory 100 may store a limited number of recent training data samples. As shown in Fig. 1, linear memory 100 serves to store recent samples from the streaming training data sources 112, 114, 116, 118 for the training pipeline. Specifically, incoming streaming data is seamlessly integrated into the training data buffer 102 with memory size N, e.g., replacing the oldest samples in the training data buffer 102 according to a first-in-first-out update rule. Schematically illustrated, the newest training data samples stored in the streaming training data sources 112, 114, 116, 118 are understood to be present on the rightmost side of the sites. As illustrated in Fig. 1, while each of the streamingtraining data sources 112, 114, 116, 118 store training data samples, only the most recently added training data samples are retrieved for storage on the training data buffer 102. Specifically, Fig. 1 illustrates schematically that the most recent training data samples are stored in site B 114 and site D 118, and these training data samples are drawn for inclusion (illustrated by down arrows) in the training data buffer 102. Older training data samples, from any of the streaming training data sources 112, 114, 116, 118, may have previously been selected for inclusion in the training data buffer 102, but are not shown as being selected at the time of the selection of the more recent training data illustrated in Fig. 1. According to various embodiments, whenever one or more new data sample is stored in any of the streaming training data sources 112, 114, 116, 118, such data sample(s) may be passed to the training data buffer 102 to be stored on a first-in-f irst-out basis.

[0031] At any time, one or more mini-batches x of training data can be generated by random sampling from the training data buffer 102 and used to train the medical machine learning system 104. This eliminates the need for the training pipeline to revisit previous samples, allowing various embodiments to leverage streaming data sources potentially of infinite length. Further, linear memory 100 facilitates the reuse of samples through multiple sampling instances, thereby reducing the overall data cost. As a result, data usage is determined by sampling rate S, which is indicative of the size of the subsets of the training data buffer that are used to train the medical machine learning system 104. By way of non-limiting example, for minibatch sizes of one, the sampling rate S may be defined as the ratio between the number of mini-batches and those acquired from the streaming source.

[0032] Fig. 2 is a schematic diagram illustrating dynamic memory 200, according to various embodiments. Fig. 2 shows the same medical machine learningsystem 104 hosted on the computer as is shown and described in reference to Fig. 1. As in Fig. 1, the medical machine learning system 104 is trained using a training data buffer 102 that stores select training data samples. Also as in Fig. 1, the training data samples stored in the training data buffer 102 are selected from a repeatedly updated stream of training data obtained though the network 106 from the plurality of online streaming training data sources 112, 114, 116, 118. As in Fig. 1, one or more minibatches may be selected from the training data buffer 102, retrieved over the network 106, and used to train the medical machine learning system 104.

[0033] To illustrate the difference between dynamic memory 200 as shown and described in reference to Fig. 2 and the prior art, a description of a typical medical data follows. Medical data obtained from hospitals often exhibit duplicated patterns, e.g., due to similar sampling strategies and / or data distribution. Initially, distributions within a period of time may adhere to the same scanning protocol and annotation policy, resulting in a high degree of duplication. However, these distributions may shift over time. This varying nature stands in contrast to the static data used in the prevalent training paradigm, where models are trained on single, meticulously curated datasets. A continual data stream in online learning setups is thus inconsistent with the prevalent training paradigm.

[0034] Accordingly, some embodiments utilize dynamic memory 200, which may be incorporated into linear memory (e.g., linear memory 100 as shown and described in reference to Fig. 1) by retaining only unique or essentially unique samples, thus actively reducing duplication. Consider the training data buffer B 102 containing N samples, each with a corresponding embedding, e.g., derived feature vector. For purposes of notation, an embedding corresponding to a sample xt is denoted zt. According to some embodiments, for adding a new sample xt to B 102, asimilarity score, such as cosine similarity, may be calculated between respective feature vectors for each pair of samples in B 102. Pairwise similarity scores for samples already present in B 102 may have been previously stored; pairwise similarity scores for newly-added samples may be determined upon addition of such samples to B 102. The pairwise similarity scores are then used to identify a sample xi* with that is most duplicative of other samples in B, where i* is the index of a sample in B 102 whose embedding has a maximal similarity among embeddings zjand zk, for j, k e B (either of j or k may be selected as i* in this example). Thus, dynamic memory 200 may be used to remove the most similar samples from B 102 at each stage after new samples have been added. Note that the removed samples may or may not include one or more of the newly-added samples. Some embodiments employ a moving average to track embeddings over iterations. Note that because each sample may represent a small region of a medical image after data augmentation, rather than the entire input, its embedding may show significant differences. By removing duplicative samples from the training data buffer B 102, the correlation between samples decreases, thereby increasing the data diversity in the training data buffer B 102.

[0035] Fig. 3 is a schematic diagram illustrating selective memory 300, according to various embodiments. In Fig. 3, the training data buffer 102, the medical machine learning system 104, the network 106, and the online streaming training data sources 112, 114, 116, 118 are as shown and described herein in reference to Figs. 1 and 2.

[0036] To illustrate the difference between selective memory 300 as shown and described in reference to Fig. 3 and the prior art, an analogy to human learning follows. In human learning, and in contrast to the naive technique of simple rote repetition, deliberate practice involves identifying the most relevant specific exercises to aid inunderstanding. Inspired by deliberate practice in humans, according to various embodiments, selective memory 300 may be used to move machine learning beyond its traditional passive role of processing a fixed set of data. Specifically, selective memory 300 prioritizes samples based on their significance, constructing a training data buffer 102 that emphasizes the most challenging samples while discarding the easier ones. Accordingly, some embodiments focus on samples with greater uncertainty and utilize structures that carry heavier penalties.

[0037] According to various embodiments, selective memory 300 may be implemented using a quantification of significance for each sample. The significance measure for a sample may represent or incorporate an error in an inference produced by the medical machine learning system 104 for the sample. Based on a current inference ability of the medical machine learning system 104, the amount of difficulty (e.g., in terms of error) that the medical machine learning system 104 has in processing a new sample may be quantified and used to implement selective memory 300. For example, for a medical machine learning system that performs regression, the significance of a sample may include a difference between an inference for the sample and the ground truth for the sample. As another example, for a medical machine learning system that performs classification, the significance of a sample may include a discrete error quantification, such as an entropy determination. As yet another example, a detailed description of significance for sample images is presented further below in the context of a medical machine learning system that performs segmentation.

[0038] Thus, Fig. 3 schematically illustrates that although all of the online streaming training data sources 112, 114, 116, 118 include recent samples (per linear memory) and also unique samples (per dynamic memory), only Site C 316 includesrecent unique samples that are also significant, per selective memory 300. Therefore, although all of the recent samples may be incorporated into the training data buffer 102, only the recent samples from Site C 316 that are also unique and significant may be retained in the training data buffer 102, while the other recent samples may be discarded from the training data buffer 102 immediately after they are incorporated according to the process described herein.

[0039] For purposes of illustration rather than limitation, a description of selective memory 300 in the context of medical image segmentation follows, by way of non-limiting example. In general, images, such as medical images, include a number of pixels. The process of segmenting an image includes labeling some or all pixels in the image. For example, some or all pixels in a medical image may assigned labels indicative of segmentation classes such as, again by way of non-limiting example, specific organs, tumor cells, background, etc. The labels may be electronically stored in associated with the image.

[0040] For selective memory 300 for automated medical image segmentation performed by the medical machine learning system 104 to classify pixels in medical images into one or more classes, the following may be used to quantify a significance (e.g., relevance, priority) of a given sample, which in the case of segmentation comprises a (two-dimensional or three-dimensional) image. In general, the significance of a sample image for purposes of automated machine learning segmentation may be quantified as a weighted sum of class entropy values, where the sum is over the segmentation classes present in the image, where each entropy value may be a binary cross entropy for a respective class, and where the weights indicate class penalties that assign smaller classes more weight. In more detail, for a given sample image, the weights and entropy values may be determined as follows. For thegiven sample image x, and for a class c in the ground truth set of classes C represented in the given image, the weight associated to the class c may be determined as a penalty acaccording to the following formula, by way of non-limiting example:fl, for 0 < <2C< 1ar= \, where cr= norm T. * C ac, otherwise\ |scl /

[0041] In the above equation, the sum is over the classes in the ground truth segmentation of the given image, |,| is the size of the class / , and |SC| is the size of the class c. The norm is a normalization of the various weights such that they lie between 0 and 1. The weights acfor c e C are then applied to a weighted sum of entropies over the classes in the sample image x. By way of non-limiting example, a binary cross-entropy may be used for each class, based on a current automated segmentation y of the sample image x performed by the medical machine learning system 104.

[0042] In general, each entropy value represents a measure of uncertainty or unpredictability associated with its associated segmentation class. Each entropy value quantifies the amount of information contained in the class, and is inspired by the idea of deliberate practice to identify and strategically target difficult problems.

[0043] Thus, a quantification of the significance Ux of the given sample image x for purposes of automated machine learning segmentation may be expressed as follows, by way of non-limiting example:Ux=acE(c)cec

[0044] In the above equation, E(c) represents the entropy associated with the class c, and acfor c e C are as described above. By way of non-limiting example, E(c) may be LBCE, the binary cross entropy for the class c.

[0045] To implement selective memory according to various embodiments, after respective significance values have been determined for each sample in the training data buffer 102, including for any newly added samples from the online streaming training data sources 112, 114, 116, 118, a number of samples are discarded from the training data buffer 102. For example, the same number of samples may be discarded as have been newly added, and the samples may be discarded immediately after the newly added samples have been incorporated and their significance values determined. The samples to be discarded may be selected according to linear memory, dynamic memory, and selective memory. In more detail, the samples to be discarded may be selected, using linear memory and dynamic memory, from among the most significant samples according to the determined significance values. For example, a parameter K may be chosen for an embodiment, where the discarded samples are selected from among the bottom K% of samples in the training data buffer 102, according to significance values. Thus, for example, an embodiment may apply an LBCE sum weighted according to class penalty to calculate significance Uxfor each sample. The embodiment may only discard samples from among those under the top K% with respect to significance when discarding samples based on a pairwise similarity comparison among all samples x.xj e BK, where BK= B\TopK{[ / x}, wherein “\” signifies set minus, and where TopK{[ / x} represents the top K% of sample with respect to significance.

[0046] This disclosure proceeds with a description of a non-limiting example implementations by the inventors, as well as test results regarding the implementations. The implementations were used to generate and maintain training data buffers, which were used to train medical image segmentation machine learning systems for 3D CT volumes.

[0047] To simulate online streaming training data sources, the inventors used two large-scale CT datasets, including a single-site private dataset (detailed in Table 1) and a sequential-site dataset (detailed in Table 2). For adapting partially labeled sub-datasets, the implementations adopted the CLIP-Driven Universal Model for multi-organ and tumor segmentation. U-Net was used as the backbone for the segmentation machine learning systems, which were trained with a default training setting with binary cross-entropy loss and the Dice loss. The Dice score was adopted as the metric to evaluate the results, which are presented in Table 1 and Table 2 for various specific anatomical features. The various implementations (but not the prevalent training paradigm used for comparison) were retrained using 100 randomly selected mini-batches of size of 1. Note that the prevalent training paradigm, which was used for comparison purposes, trains models repeatedly with 100 epochs, each using the entire contents of the respective dataset, which is generally incompatible with the streaming setting simulated for the implementations.Single-Site DatasetStrategy Linear Linear Linear Prevalent Memory Memory MemorySampling Rate S 100 100 100 - Buffer Size / V 64 128 256 - Aorta 0.8865 0.8893 0.8890 0.8848 Right Adrenal Gland 0.7530 0.7534 0.7501 0.7491 Left Adrenal Gland 0.7001 0.6980 0.6962 0.6994 Celiac Trunk 0.5403 0.5428 0.5485 0.5610 Colon 0.6915 0.6946 0.6992 0.7196Duodenum 0.7907 0.7903 0.7878 0.7896 Gall Bladder 0.8866 0.8888 0.8860 0.8886 Postcava 0.8129 0.8088 0.8161 0.8164 Right Kidney 0.9535 0.9536 0.9551 0.9527 Left Kidney 0.9473 0.9477 0.9475 0.9460 Liver 0.9715 0.9712 0.9717 0.9708 Pancreas 0.8699 0.8683 0.8703 0.8688 Intestine 0.6047 0.6088 0.6144 0.6375 Spleen 0.9664 0.9661 0.9663 0.9641 Stomach 0.9512 0.9509 0.9506 0.9453 Average Dice 0.8217 0.8222 0.8225 0.8260fable 1

[0048] The single-site dataset detailed in Table 1 and summarized in Table 3 included abdominal CT scans with per-pixel annotations, encompassing 15 classes of abdominal organs, all sourced from a single hospital with a consistent distribution. The dataset was split randomly into 2,101 training cases and 516 testing cases. Dice scores for segmenting various anatomical features for linear memory with three different buffer sizes are shown, as well Dice scores for the prevalent prior art technique based on 100 epochs.Sequential Site Data SetStrategy LM DM SM SM SM Prevalent Sampling Rate S 100 100 100 100 100 - Buffer Size / V 128 128 128 128 128 - Top K% - - 12.5 25 50 -Spleen 0.9706 0.9543 0.9553 0.9449 0.9542 0.9268 Right Kidney 0.8961 0.9236 0.9239 0.9202 0.9122 0.9189 Left Kidney 0.8972 0.9172 0.9145 0.9138 0.9052 0.9149 Gall Bladder 0.2205 0.6747 0.5367 0.5906 0.6325 0.3671 Esophagus 0.0004 0.1588 0.4376 0.4044 0.4537 0.0395 Liver 0.9615 0.9673 0.9668 0.9665 0.9632 0.9630 Stomach 0.5802 0.7705 0.7313 0.7890 0.7467 0.7785 Aorta 0.4895 0.6063 0.6751 0.5687 0.6633 0.7676 Postcava 0.2098 0.5373 0.6191 0.5444 0.5239 0.6433 Vein 0.0000 0.0370 0.0000 0.0000 0.0000 0.2310 Pancreas 0.6669 0.8028 0.8249 0.8112 0.7766 0.7376 Right Adrenal Gland 0.0000 0.0000 0.5830 0.5943 0.4307 0.0000 Left Adrenal Gland 0.0000 0.0000 0.5625 0.5126 0.4716 0.0000 Duodenum 0.1204 0.4293 0.4000 0.4987 0.5251 0.3599 Hepatic Vessel 0.4678 0.5571 0.5769 0.5524 0.5731 0.5063 Right Lung 0.7002 0.4919 0.5432 0.6112 0.5529 0.7616 Left Lung 0.8771 0.6187 0.7965 0.7162 0.7859 0.9102 Colon 0.0008 0.0003 0.0185 0.0352 0.0187 0.4986 Intestine 0.0000 0.0849 0.1081 0.2765 -.3648 0.4610 Rectum 0.0000 0.0000 0.0000 0.0000 0.0102 0.0000 Bladder 0.6383 0.5397 0.7316 0.7701 0.7722 0.7517 Prostate 0.0009 0.0923 0.1066 0.2276 0.0471 0.0000 Left Head of Femur 0.0000 0.0000 0.0000 0.0772 0.0000 0.0000 Right Head of Femur 0.0000 0.0000 0.0000 0.0690 0.0000 0.4190Celiac Trunk 0.0000 0.0000 0.0000 0.0000 0.0000 0.0000 Kidney Tumor 0.2417 0.2936 0.2483 0.3258 0.3204 0.3781 Kidney Cyst 0.0268 0.0059 0.0729 0.2920 0.0056 0.1751 Liver Tumor 0.3579 0.5753 0.5755 0.6059 0.5817 0.6219 Pancreas Tumor 0.1643 0.2823 0.2957 0.3435 0.3560 0.2058 Hepatic Vessel Tumor 0.5717 0.6659 0.6516 0.6618 0.6713 0.5930 Lung Tumor 0.1910 0.184 0.2862 0.1981 0.1516 0.3233 Colon Tumor 0.2377 0.1797 0.2751 0.2874 0.3106 0.1667 Tumor Average Dice 0.2559 0.3125 0.3436 0.3878 0.3426 0.3520 Organ Average Dice 0.3471 0.4066 0.4805 0.4958 0.4834 0.4783 Average Dice 0.3272 0.3860 0.4505 0.4722 0.4525 0.4506Table 2

[0049] Table 2 presents Dice scores for segmenting various anatomical features for linear memory (LM), dynamic memory (DM), and selective memory (SM), the latter with three different K% values, as well Dice scores for the prevalent prior art technique based on 100 epochs. The sequential-site dataset detailed in Table 2 and summarized in Table 4 included 16 partially labeled sub-datasets (Dd, cfe[1, 16]), collectively offering 32 classes. This combination of sub-datasets incorporated various annotation policies and scanning protocols, mirroring the varying distributions of a typical data stream. The data splitting protocol included 2,100 training cases and 583 testing cases. To simulate a real-world example, the data were not shuffled. As shown in Table 2, selective memory significantly outperformed other techniques on complex structures such as the Esophagus and small structures such as the Adrenal Gland.

[0050] Part of testing the implementations included testing the effectiveness of linear memory to generate training data buffers from the simulated continual datastream, and using the training data buffers to train segmentation machine learning systems. As shown in Table 3, the machine learning systems were trained both with and without linear memory on the single-site dataset with a consistent distribution.Single Site DatasetStrategy Linear Memory Linear Memory Linear Memory Repeatedly Epochs - - - 100 Sampling Rate S 100 100 100 - Buffer Size N 64 128 256 - Average Dice 0.8217 0.8222 0.8225 0.8260Table 3

[0051] Test results indicated that linear memory notably enhanced data efficiency in the continual data stream setting, without the need to revisit old data. Additionally, under the same amount of updates, the linear memory with a sampling rate S = 100 achieved similar performance to the prevalent training paradigm. Table 3 shows that by increasing training data buffer size N, the performance raised slightly due to increasing samples in the memory. However, this enhancement came with an increased memory overhead. Consequently, the training buffer size was set as N = 128 for a cost-accuracy trade-off for further tests of dynamic memory and selective memory embodiments.

[0052] The results illustrated in Table 3 demonstrate that linear memory implementations trained on continual data streams can achieve comparable performance to the prevalent training paradigm, which trains models repeatedly with 100 epochs. Moreover, linear memory allows for training without the need to revisit old data, thereby enhancing data efficiency.

[0053] Further testing using the example implementations demonstrated that dynamic memory can solve the prior art problem of catastrophic forgetting. Compared to the single-site dataset, the sequential-site dataset featured varying distributions in scanning protocol, annotation policy, and slicing scope. This variability led to significant performance degradation, as illustrated in Table 4. While naive linear memory can somewhat mitigate this issue, it did not match the performance of the prevalent training paradigm (32.7% vs. 45.1%) and still experienced substantial forgetting. Dynamic memory, by contrast, boosted performance by 6% and effectively mitigated catastrophic forgetting, as shown and described in reference to Fig. 4.Sequential Site DatasetStrategy LM DM SM SM SM Repeatedly Epochs - - - - - 100 Sampling Rate S 100 100 100 100 100 - Buffer Size / V 128 128 128 128 128 - Top K% - - 12.5 25 50 - Tumor Average Dice 0.2559 0.3125 0.3436 0.3878 0.3426 0.3520 Organ Average Dice 0.3471 0.4066 0.4805 0.4958 0.4834 0.4783 Average Dice 0.3272 0.3860 0.4505 0.4722 0.4525 0.4506Table 4

[0054] As shown in Table 4, under the varying distributions in the streaming source, Dynamic memory (DM) and Selective Memory (SM) allow for the identification of the significant samples and thereby enhanced the segmentation performance.

[0055] Fig. 4 presents charts 402, 404, 406 illustrating catastrophic forgetting during repeated retraining among machine learning systems that incorporate linear memory, dynamic memory, and selective memory, according to exampleimplementations. To evaluate forgetting, the inventors calculated the relative Dice drop after training on the incoming sub-datasets. As shown in Fig. 4, chart 402 illustrates relative Dice drop after Ds, chart 404 illustrates relative Dice drop after D12, and chart 406 illustrates relative Dice drop after D13. As shown in Fig. 4, linear memory showed a performance decline on Dd when learning from the incoming subdatasets. Both dynamic memory and selective memory store samples from previous subdatasets, thereby alleviating forgetting observed with linear memory. In particular, dynamic memory demonstrated considerable improvements, as illustrated in Fig. 4 and as further shown and described herein in reference to Fig. 5.

[0056] Fig. 5 presents charts 502, 504 illustrating the diversity of retained samples during repeated retraining, according to example implementations. As shown in chart 504, compared to dynamic memory, selective memory stores the high uncertainty samples of D6 when training on D7. This illustrates that selective memory can identify challenging but potentially duplicated samples that are ignored by dynamic memory. By deliberately targeting the samples with higher uncertainty, selective memory significantly increases the performance that even outperforms the prevalent training paradigm (47.2% vs. 45.1%; see Table 4) and is able to identify small structures such as Adrenal Gland.

[0057] Fig. 6 is a flowchart for a method 600 of updating a training data buffer based on additional training data samples from a repeatedly updated stream of training data, according to various embodiments. The method 600 may implement one or more of linear memory (e.g., as shown and described herein in reference to Fig. 1), dynamic memory (e.g., as shown and described herein in reference to Fig. 2), and selective memory (e.g., as shown and described herein in reference to Fig. 3). The method 600 may be implemented repeatedly, e.g., as part of a method of repeatedlyre-training a medical machine learning system using a repeatedly updated stream of training data, such as is shown and described herein in reference to Fig. 7. By way of non-limiting example, the training data buffer may be implemented in persistent memory, such as a hard disk drive, by way of non-limiting example.

[0058] The method 600 may be used for a repeatedly updated stream of training data. For example, the method 600 may be implemented after, e.g., in response to, one or more new training data samples being added to one or more online streaming training data sources. For example, the method 600 may be implemented based on linear memory, as shown and described herein in reference to Fig. 1.

[0059] At 602, the method 600 includes adding a number (one or more) of the new training data samples to the training data buffer. The training data sample or samples may be downloaded from the one or more online streaming training data sources over a network such as the internet. According to some embodiments, the same number of training data sample(s) may be added to the training data buffer as were added to the online streaming training data sources, e.g., since a prior execution of actions 602, 604, and 606; that is, all of the new samples from the online streaming training data source(s) may be added to the training data buffer. According to some embodiments, a different number of training data sample(s) may be added to the training data buffer in comparison to the new training data samples added to the online streaming training data sources.

[0060] At 604, the method 600 includes determining respective significance values for respective training data samples in the training data buffer. The significance values may be determined as shown and described herein in reference to Fig. 3. By way of non-limiting example, for machine learning segmentation, the significance value of a medical image may be determined as a weighted sum of class entropyvalues, where the sum is over the segmentation classes present in the image, and where the weights indicate class penalties that assign larger classes less weight. Note that the actions of 604 may be spread out over multiple steps, e.g., the actions of 604 may include computing significance values for additional training data samples and storing them in a comprehensive dictionary of training data sample significance values for the training data samples in the training data buffer.

[0061] At 606, the method 600 includes discarding one or more training data samples from the training data buffer corresponding to the number of additional training data sample(s). According to some embodiments, the same number of training data samples may be discarded as were added to the training data buffer at 602, e.g., to maintain a constant size of the training data buffer. According to some embodiments, a different number of training data samples may be discarded, e.g., such that the size of the training data buffer is allowed to vary. The samples to be discarded may be selected as shown and described herein in reference to Fig. 2 (dynamic memory), and / or Fig. 3 (selective memory). For example, the samples to be discarded may be selected based on a pairwise similarity comparison of training data samples in the training data buffer, as shown and described herein in reference to Fig. 2. As another example, in addition, or in the alternative, the samples to be discarded may be selected based on significance values associated with the samples in the training data buffer, as shown and described herein in reference to Fig. 3.

[0062] Fig. 7 is a flowchart for a method 700 of repeatedly re-training a medical machine learning system using a repeatedly updated stream of training data, according to various embodiments. The medical machine learning system may be any medical machine learning system, e.g., as shown and described herein in reference to Figs. 1 -3. By way of non-limiting example, the medical machine learningsystem may be a medical image segmentation system. The repeatedly updated stream of training data may be obtained from one or more online streaming training data source.

[0063] At 702, the method 700 includes training a medical machine learning system using training data samples. Thus, relative to the actions that follow 702, the medical machine learning system may have been previously trained using previous training data samples. The training data samples of 702 (as well as of 704) may include any of a variety of medical data, such as, by way of non-limiting example, medical images (e.g., two-dimensional or three-dimensional medical images), clinical medical data, and / or continuous medical data.

[0064] At 704, the method 700 includes acquiring additional training data samples from one or more online streaming training data source. The online streaming training data source(s) may be as shown and described herein in reference to Figs. 1-3, by way of non-limiting example. According to some embodiments, the additional training data samples may be added upon detecting that one, or a predetermined number greater than one, additional sample(s) have been included into the online streaming training data source(s). According to various embodiments, in addition or in the alternative, the additional training data samples may be added periodically, intermittently, or upon direction of a user.

[0065] At 706, the method 700 includes updating a training data buffer based on the additional training data samples. The actions of 706 may be performed as shown and described herein in reference to Fig. 6, by way of non-limiting example.

[0066] At 708, the method 700 includes retraining the medical machine learning system with at least one mini-batch from the training data buffer. The mini-batch or batches may include all or part of the contents of the training data buffer. Accordingto some embodiments, the training may include passing the mini-batch or batches through the trained machine learning system once, by way of non-limiting example.

[0067] After the actions of 708, the method 700 may include repeating the acquiring of 704, the updating of 706, and the retraining of 708, e.g., on an ongoing basis. For example, the actions of the acquiring of 704, the updating of 706, and the retraining of 708 may be repeated based on new training data samples being added to the online streaming training data source(s), periodically, intermittently, or upon direction by a user. Thus, according to some embodiments, the medical machine learning system may be continuously retrained based on new samples becoming available upon, or after, the online streaming training data source(s) are updated.

[0068] Note that because the updating actions of 706 may depend on how the medical machine learning system performs inference on samples, e.g., as shown and described herein in reference to Fig. 3, and because the retraining actions of 708 are based on the contents of the training data buffer, the method 700 may include a two-way interplay between the contents of the training data buffer and the medical machine learning system. For example, the method 700 may include a bidirectional give-and-take between the training data buffer, which is in part defined by inferences performed by the medical machine learning system, and the medical machine learning system, which is trained according to the contents of the training data buffer.

[0069] At 710, the method 700 includes providing a patient data sample to the medical machine learning system, from which a patient output is obtained. The actions of 710 may be performed at any point during the method 700. That is, that Fig. 7 illustrates the actions of 710 following 708 are purely exemplary and non-limiting. The patient data sample may include any of a variety of data, such as medical images(e.g., two-dimensional or three-dimensional medical images), clinical medical data, and / or continuous medical data.

[0070] At 712, the method 700 includes providing an indication of the patient output. The actions of 712 may include providing an indication of the patient output by displaying on a computer monitor, sending over a network such as the internet, or storing in persistent memory, such as a medical records system.

[0071] Thus, embodiments have been disclosed that address the continuously expanding volume of public medical data. Embodiments may incorporate linear memory and dynamic memory, e.g., to address data inefficiencies while retaining significant samples to effectively accommodate varying distributions. Additionally, embodiments may include selective memory to further actively focus on significant, uncertain, challenging samples. Implementations have been shown to not only surpasses the prevalent training paradigm, but also significantly mitigate catastrophic forgetting.

[0072] Certain examples can be performed using a computer program or set of programs. The computer programs can exist in a variety of forms both active and inactive. For example, the computer programs can exist as software program(s) comprised of program instructions in source code, object code, executable code or other formats; firmware program(s), or hardware description language (HDL) files. Any of the above can be embodied on a transitory or non-transitory computer readable medium, which include storage devices and signals, in compressed or uncompressed form. Exemplary computer readable storage devices include conventional computer system RAM (random access memory), ROM (read-only memory), EPROM (erasable, programmable ROM), EEPROM (electrically erasable, programmable ROM), flash memory, and magnetic or optical disks or tapes.

[0073] Aspects of the present disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented using computer readable program instructions that are executed by an electronic processor.

[0074] These computer readable program instructions may be provided to a processor of a general-purpose computer, special purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the electronic processor of the computer or other programmable data processing apparatus, create means for implementing the functions / acts specified in the flowchart and / or block diagram block or blocks. These computer readable program instructions may also be stored in a computer readable storage medium that can direct a computer, a programmable data processing apparatus, and / or other devices to function in a particular manner, such that the computer readable storage medium having instructions stored therein comprises an article of manufacture including instructions which implement aspects of the function / act specified in the flowchart and / or block diagram block or blocks.

[0075] In embodiments, the computer readable program instructions may be assembler instructions, instruction-set-architecture (ISA) instructions, machine instructions, machine dependent instructions, microcode, firmware instructions, statesetting data, configuration data for integrated circuitry, or either source code or object code written in any combination of one or more programming languages, including an object oriented programming language such as Smalltalk, C++, or the like, andprocedural programming languages, such as the C programming language or similar programming languages. The computer readable program instructions may execute entirely on a user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server.

[0076] As used herein, the terms “A or B” and “A and / or B” are intended to encompass A, B, or {A and B}. Further, the terms “A, B, or C” and “A, B, and / or C” are intended to encompass single items, pairs of items, or all items, that is, all of: A, B, C, {A and B}, {A and C}, {B and C}, and {A and B and C}. The term “or” as used herein means “and / or.”

[0077] As used herein, language such as “at least one of X, Y, and Z,” “at least one of X, Y, or Z,” “at least one or more of X, Y, and Z,” “at least one or more of X, Y, or Z,” “at least one or more of X, Y, and / or Z,” or “at least one of X, Y, and / or Z,” is intended to be inclusive of both a single item (e.g., just X, or just Y, or just Z) and multiple items (e.g., {X and Y}, {X and Z}, {Y and Z}, or {X, Y, and Z}). The phrase “at least one of” and similar phrases are not intended to convey a requirement that each possible item must be present, although each possible item may be present.

[0078] The techniques presented and claimed herein are referenced and applied to material objects and concrete examples of a practical nature that demonstrably improve the present technical field and, as such, are not abstract, intangible or purely theoretical. Further, if any claims appended to the end of this specification contain one or more elements designated as “means for [perform]ing [a function]...” or “step for [performing [a function]...”, it is intended that such elements are to be interpreted under 35 U.S.C. § 112(f). However, for any claims containingelements designated in any other manner, it is intended that such elements are not to be interpreted under 35 U.S.C. § 112(f).

[0079] While the invention has been described with reference to the exemplary examples thereof, those skilled in the art will be able to make various modifications to the described examples without departing from the true spirit and scope. The terms and descriptions used herein are set forth by way of illustration only and are not meant as limitations. In particular, although the method has been described by examples, the steps of the method can be performed in a different order than illustrated or simultaneously. Those skilled in the art will recognize that these and other variations are possible within the spirit and scope as defined in the following claims and their equivalents.

Claims

What is claimed is:

1. A method of repeatedly re-training a medical machine learning system using a repeatedly updated stream of training data, wherein the medical machine learning system has been previously trained using previous training data samples, the method comprising:acquiring at least one additional training data sample from at least one online streaming training data source;updating a training data buffer based on the at least one additional training data sample, wherein the updating comprises:adding a number of the at least one additional training data sample to the training data buffer,determining respective significance values for respective training data samples in the training data buffer, anddiscarding at least one training data sample from the training data buffer corresponding to the number of the at least one additional training data sample, wherein the discarding comprises discarding based on a pairwise similarity comparison of training data samples in the training data buffer and based on the respective significance values; andretraining the medical machine learning system with at least one mini-batch from the training data buffer.

2. The method of claim 1, further comprising:providing a patient data sample to the medical machine learning system, from which a patient output is obtained; andproviding an indication of the patient output.

3. The method of claim 1, further comprising repeating the acquiring, the updating, and the retraining on an ongoing basis, whereby the medical machine learning system is continuously retrained based on the repeatedly updated stream of training data.

4. The method of claim 1, wherein the medical machine learning system comprises a machine learning image segmentation system.

5. The method of claim 4, wherein a respective significance value comprises a respective weighted sum of class entropy values for segmentation classes in a respective image.

6. The method of claim 5, wherein the respective weighted sum of class entropy values comprises a respective penalized weighted sum of binary cross entropy values.

7. The method of claim 1, wherein a respective significance value for a respective training data sample is based on a misclassification of the respective training data sample by the medical machine learning system.

8. The method of claim 1, wherein the acquiring comprises acquiring the at least one additional training data sample from a plurality of online streaming training data sources.

9. The method of claim 1, wherein a respective training data sample comprises a three-dimensional image.

10. The method of claim 1, wherein a respective pairwise similarity comparison comprises a cosine similarity of a respective pair of feature vectors of a respective pair of training data samples.

11. A system for repeatedly re-training a medical machine learning system using a repeatedly updated stream of training data, wherein the medical machine learning system has been previously trained using previous training data samples, the system for repeatedly re-training the medical machine learning system comprising: a non-transitory computer readable medium comprising instructions; and at least one electronic processor that executes the instructions to perform operations comprising:acquiring at least one additional training data sample from at least one online streaming training data source;updating a training data buffer based on the at least one additional training data sample, wherein the updating comprises:adding a number of the at least one additional training data sample to the training data buffer,determining respective significance values for respective training data samples in the training data buffer, anddiscarding at least one training data sample from the training data buffer corresponding to the number of the at least one additional training data sample, wherein the discarding comprises discarding based on a pairwisesimilarity comparison of training data samples in the training data buffer and based on the respective significance values; andretraining the medical machine learning system with at least one mini-batch from the training data buffer.

12. The system for repeatedly re-training the medical machine learning system of claim 11, wherein the operations further comprise:providing a patient data sample to the medical machine learning system, from which a patient output is obtained; andproviding an indication of the patient output.

13. The system for repeatedly re-training the medical machine learning system of claim 11, wherein the operations further comprise repeating the acquiring, the updating, and the retraining on an ongoing basis, whereby the medical machine learning system is continuously retrained based on the repeatedly updated stream of training data.

14. The system for repeatedly re-training the medical machine learning system of claim 11, wherein the medical machine learning system comprises a machine learning image segmentation system.

15. The system for repeatedly re-training the medical machine learning system of claim 14, wherein a respective significance value comprises a respective weighted sum of class entropy values for segmentation classes in a respective image.

16. The system for repeatedly re-training the medical machine learning system of claim 15, wherein the respective weighted sum of class entropy values comprises a respective penalized weighted sum of binary cross entropy values.

17. The system for repeatedly re-training the medical machine learning system of claim 11, wherein a respective significance value for a respective training data sample is based on a misclassification of the respective training data sample by the medical machine learning system.

18. The system for repeatedly re-training the medical machine learning system of claim 11, wherein the acquiring comprises acquiring the at least one additional training data sample from a plurality of online streaming training data sources.

19. The system for repeatedly re-training the medical machine learning system of claim 11, wherein a respective training data sample comprises a three-dimensional image.

20. The system for repeatedly re-training the medical machine learning system of claim 11, wherein a respective pairwise similarity comparison comprises a cosine similarity of a respective pair of feature vectors of a respective pair of training data samples.

Images