Bias reduction for machine learning feature selection

Abstract

A method may include splitting the dataset into holdout sets. Multiple training and testing sets may be generated by splitting, for each holdout fold in the holdout sets, a remaining plurality of samples not included in the holdout fold into a training and testing set. Multiple iterations of computational feature selection may be performed using the holdout sets and the training and testing sets. Each iteration may include training and validating a first instance of a machine learning model on a training and testing set before training, using the same samples in the training and testing set, a second instance of the machine learning model to operate on a subset of features selected during the training. Each iteration may further include validating of the first instance of the machine learning model before applying the second instance of the machine learning model to a holdout fold from the holdout sets.

Description

BIAS REDUCTION FOR MACHINE LEARNING FEATURE SELECTION CROSS REFERENCE TO RELATED APPLICATION

[0001] This application claims priority to U.S. Provisional Application No. 63 / 761,796, entitled “BIAS REDUCTION FOR MACHINE LEARNING FEATURE SELECTION” and filed on February 21, 2025, the disclosure of which is incorporated herein by reference in its entirety.TECHNICAL FIELD

[0002] The subject matter described herein relates generally to machine learning and more specifically to techniques for reducing bias when selecting input features for a machine learning model.BACKGROUND

[0003] Machine learning models may be trained to perform a variety of cognitive tasks. For example, generative models, such as generative adversarial networks (GANs), variational autoencoders (VAEs), diffusion models, and transformers, are trained to learn the underlying probability distribution of data such that new data samples can be generated from the same probability distribution. Alternatively, discriminative models, such as logistic regression models, support vector machines (SVMs), and multilayer perceptrons (MLPs), are trained to learn the boundaries between different classes (or categories) of data to determine, for each possible class (or category), the probability of a new data sample being a member of the class (or category). In the pharmaceutical and biotechnology sector, generative and discriminative models are deployed for a wide variety of applications including, for example, early-stage drug discovery, druggable target identification, biological marker (or biomarker) detection, patient stratification, disease modeling, clinical trial design, and / or the like.SUMMARYAttorney Ref: 105892.228002

[0004] Systems, methods, and articles of manufacture, including computer program products, are provided for biomarker selection with bias reduction.

[0005] In one aspect, there is provide a system for biomarker selection with bias reduction. The system may include at least one data processor and at least one memory. The at least one memory may store instructions that result in operations when executed by the at least one data processor. The operations may include: receiving a dataset including a plurality of samples, wherein each sample in the plurality of samples include a set of features; splitting the dataset into one or more holdout sets; generating a plurality of training and testing sets by at least splitting, for each holdout fold in the one or more holdout sets, a remaining plurality of samples not included in the holdout fold into a training and testing set; performing a plurality of iterations of computational feature selection using the one or more holdout sets and the plurality of training and testing sets, wherein each iteration of computational feature selection includes training and validating a first instance of a machine learning model on a training and testing set, wherein each iteration of computational feature selection further includes training, using a same plurality of samples in the training and testing set, a second instance of the machine learning model to operate on a subset of features selected during the training and testing of the first instance of the machine learning model before applying the second instance of the machine learning model to a holdout fold from the one or more holdout sets; selecting, based at least on a frequency with which each feature present in the dataset is selected across the plurality of iterations of computational feature selection, an aggregate subset of features from the dataset; and applying the aggregate subset of features to one or more downstream tasks.

[0006] In another aspect, there is provided a computer-implemented method for biomarker selection with bias reduction. The method may include: receiving a dataset including a pluralityAttorney Ref: 105892.228002 of samples, wherein each sample in the plurality of samples include a set of features; splitting the dataset into one or more holdout sets; generating a plurality of training and testing sets by at least splitting, for each holdout fold in the one or more holdout sets, a remaining plurality of samples not included in the holdout fold into a training and testing set; performing a plurality of iterations of computational feature selection using the one or more holdout sets and the plurality of training and testing sets, wherein each iteration of computational feature selection includes training and validating a first instance of a machine learning model on a training and testing set, wherein each iteration of computational feature selection further includes training, using a same plurality of samples in the training and testing set, a second instance of the machine learning model to operate on a subset of features selected during the training and testing of the first instance of the machine learning model before applying the second instance of the machine learning model to a holdout fold from the one or more holdout sets; selecting, based at least on a frequency with which each feature present in the dataset is selected across the plurality of iterations of computational feature selection, an aggregate subset of features from the dataset; and applying the aggregate subset of features to one or more downstream tasks.

[0007] In another aspect, there is provided a computer program product for biomarker selection with bias reduction. The computer program product may include a non-transitory computer readable medium storing instructions that result in operations when executed by at least one data processor. The operations may include: receiving a dataset including a plurality of samples, wherein each sample in the plurality of samples include a set of features; splitting the dataset into one or more holdout sets; generating a plurality of training and testing sets by at least splitting, for each holdout fold in the one or more holdout sets, a remaining plurality of samples not included in the holdout fold into a training and testing set; performing a plurality of iterationsAttorney Ref: 105892.228002 of computational feature selection using the one or more holdout sets and the plurality of training and testing sets, wherein each iteration of computational feature selection includes training and validating a first instance of a machine learning model on a training and testing set, wherein each iteration of computational feature selection further includes training, using a same plurality of samples in the training and testing set, a second instance of the machine learning model to operate on a subset of features selected during the training and testing of the first instance of the machine learning model before applying the second instance of the machine learning model to a holdout fold from the one or more holdout sets; selecting, based at least on a frequency with which each feature present in the dataset is selected across the plurality of iterations of computational feature selection, an aggregate subset of features from the dataset; and applying the aggregate subset of features to one or more downstream tasks.

[0008] In some variations, one or more features disclosed herein including the following features can optionally be included in any feasible combination.

[0009] In some variations, each iteration of computational feature selection includes training the first instance of the machine learning model on a plurality of training folds in the training and testing set, and validating the first instance of the machine learning model on a testing fold in the training and testing set.

[0010] In some variations, the plurality of iterations of computational feature selection includes a first iteration in which the first instance of the machine learning model is trained and tested on a first training and testing set before the second instance of the machine learning model applied to a first holdout fold from a first holdout set.

[0011] In some variations, the plurality of iterations of computational feature selection includes a second iteration in which a third instance of the machine learning model isAttorney Ref: 105892.228002 trained and tested on a second training and testing set, and a fourth instance of the machine learning model is trained using the same plurality of samples in the second training and testing set to operate on a second subset of data selected during the training and testing of the third instance of the machine learning model.

[0012] In some variations, the second iteration further includes applying the fourth instance of the machine learning model to a second holdout fold from the first holdout set.

[0013] In some variations, the second iteration further includes applying the fourth instance of the machine learning model to a second holdout fold from a second holdout set.

[0014] In some variations, an average performance of the machine learning model is determined based at least on a performance of the machine learning model applied to a testing fold in each training and testing set.

[0015] In some variations, a distribution of the performance of the machine learning model is determined based at least on a performance of the machine learning model applied to each holdout fold.

[0016] In some variations, the machine learning model assigns each feature of the plurality of features a coefficient for combining the plurality of features into an output of the machine learning model.

[0017] In some variations, the logistic regression model includes regularization to reduce, to a zero value, any coefficient whose value fails to satisfy one or more thresholds.

[0018] In some variations, a feature is selected when a coefficient associated with the feature is assigned a non-zero value by the regularization.

[0019] In some variations, the regularization comprises LI regularization, L2 regularization, and / or elastic net.Attorney Ref: 105892.228002

[0020] In some variations, the one or more downstream tasks include a new instance of the machine learning model trained to generate an output by at least operating on the aggregate subset of features.

[0021] In some variations, the output includes a classification of a patient sample to indicate one or more of disease or healthy, benign or malignant, disease subtype, disease stage, responder or non-responder, or treatment selection.

[0022] In some variations, the one or more downstream tasks includes an assay panel targeting a plurality of genes comprising the aggregate subset of features.

[0023] In some variations, the plurality of samples include a plurality of patient samples.

[0024] In some variations, the plurality of samples include a plurality of multiomics profiles.

[0025] In some variations, the set of features include one or more genes expressing a protein or a ribonucleic acid (RNA) fragment present in a biopsy specimen.

[0026] In some variations, a first plurality of iterations of computational feature selection is performed for a first holdout set.

[0027] In some variations, a second plurality of iterations of computational feature selection is performed for a second holdout set.

[0028] In some variations, the first holdout set and the second holdout set each includes a plurality of holdout folds.

[0029] In some variations, each holdout fold is associated with a plurality of training folds and a testing fold generated by splitting the remaining plurality of samples.Attorney Ref.: 105892.228002

[0030] In some variations, the machine learning model comprises one or more of a logistic regression model, a linear regression model, or a support vector machine.

[0031] Implementations of the current subject matter can include, but are not limited to, methods consistent with the descriptions provided herein as well as articles that comprise a tangibly embodied machine-readable medium operable to cause one or more machines (e.g., computers, etc.) to result in operations implementing one or more of the described features. Similarly, computer systems are also described that may include one or more processors and one or more memories coupled to the one or more processors. A memory, which can include a non-transitory computer-readable or machine-readable storage medium, may include, encode, store, or the like one or more programs that cause one or more processors to perform one or more of the operations described herein. Computer implemented methods consistent with one or more implementations of the current subject matter can be implemented by one or more data processors residing in a single computing system or multiple computing systems. Such multiple computing systems can be connected and can exchange data and / or commands or other instructions or the like via one or more connections, including, for example, to a connection over a network (e.g. the Internet, a wireless wide area network, a local area network, a wide area network, a wired network, or the like), via a direct connection between one or more of the multiple computing systems, etc.

[0032] The details of one or more variations of the subject matter described herein are set forth in the accompanying drawings and the description below. Other features and advantages of the subject matter described herein will be apparent from the description and drawings, and from the claims. While certain features of the currently disclosed subject matter are described for illustrative purposes in relation to biomarkers, it should be readily understood thatAttorney Ref: 105892.228002 such features are not intended to be limiting. The claims that follow this disclosure are intended to define the scope of the protected subject matter.DESCRIPTION OF DRAWINGS

[0033] The accompanying drawings, which are incorporated in and constitute a part of this specification, show certain aspects of the subject matter disclosed herein and, together with the description, help explain some of the principles associated with the disclosed implementations. In the drawings,

[0034] FIG. 1 depicts a system diagram illustrating an example of a feature selection system, in accordance with some example embodiments;

[0035] FIG. 2A depicts a flowchart illustrating an example of a biomarker selection process, in accordance with some example embodiments;

[0036] FIG. 2B depicts a flowchart illustrating another example of a biomarker selection process, in accordance with some example embodiments;

[0037] FIG. 3 depicts a schematic diagram illustrating an example of training, testing, and holdout folds across multiple iterations of computational feature selection, in accordance with some example embodiments;

[0038] FIG. 4 depicts a flowchart illustrating an example of a process for selecting biomarkers for machine learning enabled prediction of treatment response, in accordance with some example embodiments; and

[0039] FIG. 5 depicts a block diagram illustrating an example of a computing system, in accordance with some example embodiments.

[0040] When practical, similar reference numbers denote similar structures, features, or elements.Attorney Ref: 105892.228002DETAILED DESCRIPTION

[0041] The performance of a machine learning model, such as a generative model or a discriminative model, may be contingent upon the features used by the machine learning model to derive its output. In the context of artificial intelligence and machine learning, a feature is an attribute in the representation of an object (or a phenomenon) that describes one or more aspects of the object (or phenomenon). For example, in some cases, the representation of the object may be an n-dimensional vector containing an n quantity of numerical values corresponding to an n-quantity of features. In some cases, the individual features are variables, meaning that difference instances of the object may have different values for one or more of the n-quantity of features. A machine learning model may be trained to perform a task by learning the patterns, or the probability distribution of feature vectors with different combinations of values, that are present across multiple instances of the object. In cases where the task is classification, learning the patterns (or probability distribution) may enable the machine learning model to classify a new instance of the object based on the corresponding feature vector. Alternatively, for a generative task, learning the patterns (or probability distribution) may enable the machine learning model to generate new instances of the object whose feature vectors exhibit the same patterns (or probability distribution).

[0042] In some cases, some but not all of the features associated with the object (or phenomenon) may be relevant to the task performed by the machine learning model. Instead of every available feature associated with the object, the machine learning model may operate on a subset of all those features in order to produce high accuracy predictions. An excessive quantity of features may, in some cases, actually diminish the performance of the machine learning model due to the presence of trivial or redundant features obscuring key patterns and relationships. ForAttorney Ref: 105892.228002 example, a patient sample, such as the multi omics profile of the constituents of a biopsy specimen, may contain numerous proteins and nucleic acids (e.g., deoxyribonucleic acid (DNA), ribonucleic acid (RNA), and / or the like). Some but not all of those proteins and nucleic acids may be informative as biomarkers for discerning the presence and, in some cases, the progress (or stage) of a disease (e.g., cancer, neurodegeneration, and / or the like) in the patient.

[0043] Accordingly, in some example embodiments, the features associated with an object (or phenomenon), such as the proteins and nucleic acids identified in multiomics profile of a patient, may undergo computational feature selection in order to identify a subset of features. For example, where the object is associated with an m-quantity of features, a subset of n features (where m > n) may be selected. In some cases, the n-quantity of features may be an n-quantity of biomarkers (e.g., proteins, nucleic acids, and / or the like) used by a machine learning model to detect the presence of a disease or identifying the progress (or stage) of the disease in a patient based on the n-quantity of biomarkers. Alternatively and / or additionally, an assay panel may be configured to target these n-quantity of biomarkers specifically rather than the m-quantity of proteins and nucleic acids that may be present in each patient sample. In some cases, a first biomarker identified as relevant through computational feature selection may replace a different but correlated second biomarker in the assay panel in instances where the second biomarker fails to meet the wet lab requirements for validation. This could be the case when, for example, no assays targeting the first biomarker is available. In some cases, it may also be possible for the quantity of the first biomarker present in typical patient samples to be too low (e.g., inadequate signal-to-noise ratio (SNR)) for detection by available assays.

[0044] Feature selection, which is typically performed using data from a limited subpopulation of objects (or phenomena), may be biased when the selected features are notAttorney Ref.: 105892.228002 representative of a larger or more general population. For example, the n-quantity of biomarkers selected based on a set of patient samples may exhibit bias associated with the over- or underrepresentation of certain types of data (e.g., demographics, diseased and healthy, early and late stage, and / or the like) in that set of patient samples. Biased feature selection may diminish the performance of downstream applications, such as an assay panel targeting the n-quantity of biomarkers or a machine learning model operating on the n-quantity of biomarkers, when those downstream applications are extended to a larger, more general patient population. This may be due to the set of patient samples skewing feature selection towards omitting features that are relevant to the larger, more general patient population or including features that are irrelevant to the larger, more general patient population. Various implementations of the present disclosure reduce bias by selecting features that are identified as relevant across multiple iterations of computational feature selection, with a different instance of a machine learning model (e.g., a logistic regression model, a linear regression model, and / or a support vector machine with regularization such as LI regularization, L2 regularization, elastic net, and / or the like) being trained on a different set of training folds (or partitions) from a dataset during each iteration. As described in more details below, each iteration of computational feature selection may further include testing, using a different holdout fold from the dataset, one or more instances of the machine learning model trained to operate on a subset of features. In some case, the holdout fold for a single iteration of computational feature selection may exclude samples in the testing and training folds used during the same iteration of computational feature selection in order to best emulate real world deployment scenarios in which the machine learning model is exposed to data it has not encountered during training.Attorney Ref: 105892.228002

[0045] In some example embodiments, upon completing the multiple iterations of computational feature selection, the subset of features selected for one or more downstream tasks (e.g., a machine learning model, a targeted assay panel, and / or the like) may include those features that were selected during a threshold quantity of iterations of computational feature selection. Moreover, in some cases, the performance of the machine learning model operating on different training, testing, and holdout folds from the dataset may provide an estimate of the performance of a machine learning model operating on the subset of features when that machine learning model is deployed and applied to real world data. For example, in some cases, the performance of the machine learning model may be estimated based on the performance (e.g., quantified by the area under the receiver operator characteristic (ROC) curve (AUC)) achieved by the machine learning model across the different training folds. Furthermore, in some cases, the performance of the machine learning model may be estimated based on the performance (e.g., quantified by the area under the receiver operator characteristic (ROC) curve (AUC)) achieved by the machine learning model across the different holdout sets.

[0046] FIG. 1 depicts a system diagram illustrating an example of a feature selection system 100, in accordance with some example embodiments. Referring to FIG. 1, the feature selection system 100 may include a feature selection engine 110, a machine learning engine 120, a data store 130, and a client device 140. As shown in FIG. 1, the feature selection engine 110, the machine learning engine 120, the data store 130, and the client device 140 may be communicatively coupled via a network 150. In some cases, the data store 130 may be a database including, for example, a relational database, a NoSQL database, a columnar database, an objected-oriented database, a key -value database, a hierarchical database, a document database, a graph database, and / or the like. In some cases, the client device 140 may be a processor-basedAttorney Ref: 105892.228002 device including, for example, a workstation, a desktop computer, a laptop computer, a smartphone, a tablet computer, a wearable apparatus, and / or the like. In some cases, the network 150 may be a wired network and / or a wireless network including, for example, a local area network (LAN), a virtual local area network (VLAN), a wide area network (WAN), a public land mobile network (PLMN), the Internet, and / or the like.

[0047] In some example embodiments, the feature selection engine 110 may perform computational feature selection in order to determine, based on a dataset 135 (e.g., stored in the data store 130), a feature set 115. In some cases, the feature set 115 may include a subset of the features present in the dataset 135. If an m-quantity of features are present in the dataset 135, for example, the feature set 115 may include an n-quantity of features instead. In some cases, each feature in the feature set 115 may be an attribute in the representation of an object (or a phenomenon) that describes one or more aspects of the object (or the phenomenon). For example, in some cases, each feature in the feature set 115 may be a biomarker, which is a medical sign that is capable of being measured accurately and reproducibly to provide an objective indication of a medical state of a patient. A protein or a nucleic acid (e.g., deoxyribonucleic acid (DNA), ribonucleic acid (RNA), and / or the like) in the multi omics profile of the constituents of a patient’s biopsy sample (e.g., tissue biopsy, liquid biopsy, and / or the like) is one example of a biomarker. In some cases, while an m-quantity of biomarkers (e.g., proteins, nucleic acids, and / or the like) may be present in the patient’s biopsy sample, the feature selection engine 110 may identify a subset containing an n-quantity of biomarkers for inclusion in the feature set 115.

[0048] In some example embodiments, the feature selection engine 110 may determine the feature set 115 such that a machine learning model 125 may operate on the feature set 115 to perform a generative task or a classification task. For example, in cases where theAttorney Ref: 105892.228002 feature set 115 includes the n-quantity of biomarkers, such as the proteins and nucleic acids (e.g., deoxyribonucleic acid (DNA), ribonucleic acid (RNA), and / or the like) in the multiomics profile of a patient, the machine learning model 125 may operate on the feature set 115 to assign one or more labels classifying the patient (e.g., disease or healthy, benign or malignant, disease subtype, disease stage, responder or non-responder, treatment selection, and / or the like). Alternatively and / or additionally, the machine learning model 125 may operate on the feature set 115 when modifying the patient’s multiomics profile to generate one or more new (e.g., synthetic) multiomics profiles. In some cases, the biomarkers (e.g., proteins, nucleic acids, and / or the like) in the feature set 115 may also form an assay panel targeting these biomarkers. For instance, in some cases, a first biomarker identified as relevant through computational feature selection may replace a different but correlated second biomarker in the assay panel in instances where the second biomarker fails to meet the wet lab requirements for validation.

[0049] FIG. 2A depicts a flowchart illustrating an example of a process 200 for computational feature selection with bias reduction, in accordance with some example embodiments. Referring to FIGS. 1 and 2A, the process 200 may be performed by the feature selection engine 110, for example, to generate the feature set 115 for use by the machine learning model 125 to perform a generative task, a classification task, and / or the like. For example, in instances where the feature set 115 includes biomarkers (e.g., proteins, nucleic acids, and / or the like), the machine learning model 125 may operate on the feature set 115 to assign one or more labels classifying the patient (e.g., disease or healthy, benign or malignant, disease subtype, disease stage, responder or non-responder, treatment selection, and / or the like). Alternatively and / or additionally, the machine learning model 125 may operate on the feature set 115 when modifying the patient’s multiomics profile to generate one or more new (e.g., synthetic) multiomics profiles.Attorney Ref: 105892.228002 In some cases, the feature set 115 may also be generated for use as an assay panel targeting the biomarkers (e.g., proteins, nucleic acids, and / or the like) included in the feature set 115.

[0050] At 202, a dataset including a plurality of samples is received. In some example embodiments, the dataset may include a plurality of patient samples. Moreover, in some cases, each sample, such as patient sample, may include multiple features. In some cases, each feature may be an attribute of a physiological state and / or a pathological state of a patient. For example, in some cases, each patient sample in the dataset may include the multiomics profile of the constituents of a biopsy specimen (e.g., liquid biopsy specimen, tissue biopsy specimen, and / or the like). In some cases, the multiomics profile may be the multiomics profile of the cargo carried by extracellular vesicles (or certain subpopulations of extracellular vesicles) isolated from the biopsy specimen. In some cases, the multiomics profile of the cargo may be a genomics and proteomics profile of the proteins (e.g., splice variants, isoforms, post translational modifications (PTMs), and / or the like), ribonucleic acid (RNA) (e.g., messenger RNA (mRNA), small noncoding RNA (sncRNA), long non-coding RNA (IncRNA), circular RNA (circRNA), and / or the like), and deoxyribonucleic acid (DNA) (e.g., mutations, epigenetics, and / or the like) present in the cargo of the extracellular vesicles (or subpopulations of extracellular vesicles). In some cases, the “features” in the multiomics profile may correspond to the genes encoding the proteins and / or ribonucleic acid (RNA) fragments present.

[0051] At 204, at least a portion of the dataset is preprocessed. In some example embodiments, the preprocessing of the dataset may include preprocessing the samples, such as the multiomics profile of each patient, to remove batch effect (e.g., due to technical variations arising from instrument fluctuations or laboratory controlled variables) and to identify the features contained therein. For example, in some cases, the preprocessing of the dataset may includeAttorney Ref: 105892.228002 identifying one or more genes encoding at least a portion of the proteins and RNA fragments (or molecules) present in each patient sample such that each physiological state of the corresponding patient is represented by a set of features, such as an m quantity of genes encoding at least a portion of the constituent proteins and RNA fragments (or molecules). As described in more details below, a subset of these features, such as an n quantity of genes, may be identified through computational feature selection as biomarkers for one or more downstream tasks, such as for use by a machine learning model or in a targeted assay panel.

[0052] In some example embodiments, the preprocessing of the dataset may include preprocessing the genomics data present in the dataset. In some cases, the genomics data may include RNA sequencing (RNA-seq) data. In some cases, the RNA sequencing (RNA-seq) data may be raw sequence reads, which are trimmed, aligned, and mapped to corresponding genes (e.g., the gene expressing each RNA sequence). In this context, the term “read” refers to the inferred sequence of nucleotide bases in each RNA fragment or molecule determining using a sequencing technique (e.g., short-read sequencing, long-read sequencing, and / or the like). In some cases, the read count for an RNA sequence mapped to an individual gene, which corresponds to the number of nucleotides in the RNA sequence, may be scaled by the length of the gene. It should be appreciated that longer genes may produce higher read counts than shorter genes at the same sequencing depth (e.g., the number of times a specific nucleotide base is read during the sequencing process). Because the expression level of a gene is measured by counting the number of reads mapped to the gene, scaling the read count for an RNA sequence by the length of the corresponding gene may remove the bias introduced by gene length in the calculation of gene expression levels. In some cases, in addition to gene-length scaling for RNA sequences, the preprocessing may also include scaling to reduce the effect of larger counts (read counts for RNAAttorney Ref: 105892.228002 or intensities for protein) such that ‘smaller’ intensities can be as important as larger ones. In some cases, genes without any reads may be assigned zero reads. In some cases, genes may be filtered based on their respective read counts to exclude those that fail to satisfy one or more criteria, such as having a minimum read count across a threshold quantity patient samples for an indication. In some cases, the read count data may undergo normalization prior to differential gene expression (DGE) analysis (e.g., to identify genes that expressed differently across two or more patient samples) and functional enrichment analysis (e.g., to annotate and contextualize the genes identified through differential gene expression (DGE) analysis). In some cases, the read count data may also be normalized with Variance Stabilizing Transformation (VST), assessed with principle component analysis (PCA), and batch corrected (e g., to remove technical variations arising from batch effects).

[0053] In some example embodiments, the preprocessing of the dataset may include preprocessing the proteomics data present in the dataset. In some cases, the proteomics data may include peptide-level data, which are data at the sub-protein level that can provide insights into sub-protein changes such as splice variants, isoforms, post-translational modifications, and / or the like. In some cases, the peptide-level data may be mapped to protein level data. In some cases, the read counts for proteins may undergo imputation to correct for non-biological zero values, which are spurious zero values resulting from technical or sampling errors. Moreover, in some cases, the protein level data may be filtered to exclude proteins that fails to satisfy one or more criteria, such as a minimum abundance level in a threshold quantity of patient samples for an indication. In some cases, the protein level data may be assessed with principle component analysis (PCA) and batch corrected (e.g., to remove technical variations arising from batch effects). In some cases, zero expression values may be imputed with a mixed strategy. ForAttorney Ref: 105892.228002 example, in some cases, if no trace of a protein encoded by a gene is present the dataset, the expression level of that protein may be imputed to a certain minimum value. Alternatively, the expression level of a protein may be imputed based on the expression levels of similar proteins, for example, using : -Nearest Neighbor (kNN). In some cases, upon normalization and imputation, the proteomics data may also be normalized, for example, using Variance Stabilization Normalization (VSN)4.

[0054] At 206, an aggregate subset of features present in the dataset is selected by at least applying, over multiple iterations of computational feature selection, a machine learning model to operate on different combinations of training, testing, and holdout folds derived from the dataset. In some example embodiments, each patient sample in the dataset may include a plurality of features representative of the physiological state of the corresponding patient. For example, in some cases, each patient sample may include an m quantity of genes, each of which encoding a protein or an RNA fragment (or molecule) present in a biopsy specimen of the corresponding patient. In some cases, the dataset may undergo computational feature selection in order to identify a subset of features, such as an n quantity of genes, for use as biomarkers in one or more downstream tasks. Various implementations of the computational feature selection described here may reduce the bias that arise from the over- or under-representation of certain types of data in the dataset. For instance, with conventional feature selection methodologies, if the dataset includes patient samples that are predominantly from one patient demographics, the n quantity of genes that are selected therefrom may be relevant to that particular patient demographics meaning that a machine learning model relying on these n quantity of genes is likely to perform poorly when applied to other patient demographics. As described in more details below, the bias in feature selection may be reduced by training and validating a machine learning model over multipleAttorney Ref: 105892.228002 iterations of computational feature selection, with each iteration training and validating one or more instances of the machine learning model on a different training and testing set from the dataset before one or more instances of the machine learning model trained using the same data from the training and testing set to operate on a subset of features are tested on a holdout fold that excludes any samples from the folds in the training and testing set.

[0055] At 208, the aggregate subset of features are applied to one or more downstream tasks. In some example embodiments, the dataset may undergo computational feature selection in order to identify a subset of features, such as an n quantity of genes, for use as biomarkers in one or more downstream tasks. For example, in some cases, the n quantity of genes may be used by a machine learning model trained to determine, based at least on an expression level of each of the n quantity of genes detected in a biopsy specimen, a classification such as disease or healthy, benign or malignant, disease subtype, disease stage, responder or nonresponder, treatment selection, and / or the like. In some cases, the performance of the machine learning model, such as an area under the curve (AUC) confidence interval indicative of the uncertainty in the output of the machine learning model, may be estimated based on the performance of different instances of the machine learning model operating on different testing folds and holdout folds from the dataset. Moreover, in some cases, the n quantity of genes may be included in an assay panel targeting the corresponding proteins or RNA fragments (or molecules). For instance, in some cases, a first gene identified as relevant through computational feature selection may replace a different but correlated second gene in the assay panel in instances where the second gene fails to meet the wet lab requirements for validation.

[0056] FIG. 2B depicts a flowchart illustrating an example of a process 250 for computational feature selection with bias reduction, in accordance with some exampleAttorney Ref: 105892.228002 embodiments. Referring to FIGS. 1 and 2A-B, the process 250 may be performed by the feature selection engine 110, for example, to generate the feature set 115 for use by the machine learning model 125 to perform a generative task, a classification task, and / or the like. In some cases, the process 300 may implement operation 204 of the process 200 shown in FIG. 2A. For example, in some cases, the process 250 may include the feature selection engine 110 splitting the dataset 135 into multiple different folds for training and validating different instances of the machine learning model 125. In some cases, the feature selection engine 110 may split the dataset 135 into one or more holdout sets, each of which including multiple holdout folds such as, for example, a first holdout fold, a second holdout fold, and / or the like. As described in more details below, the holdout fold used during each iteration of computational feature selection may exclude samples present in the folds in the training and testing set used for that iteration. That is, in some cases, the features selection engine 110 may, for each iteration of computational feature selection, split the samples that were not included in the holdout set into a training and testing set that includes multiple training and testing folds. For instance, in some cases, a first iteration of computational feature selection may include an instance of a machine learning model trained to operate on a first subset of features being tested using the first holdout fold after another instance of the model is trained and tested on a first training and testing set that excludes samples from the first holdout fold in order to identify the first subset of features. Furthermore, a second iteration of computational feature selection may include yet another instance of the machine learning model trained to operate on a second subset of features being tested using the second holdout fold after another instance of the machine learning model is trained and tested on a second training and testing set that excludes samples from the second holdout fold in order to identify the second subset of features.Attorney Ref.: 105892.228002

[0057] At 252, a dataset is split into one or more holdout sets. In some example embodiments, the dataset may be split into one or more holdout sets, each of which including a k quantity of holdout folds. For example, in some cases, the dataset may be split into at least one holdout set containing, for example, 5 holdout folds, 10 holdout folds, and / or the like. In some cases, the dataset may include a j quantity of patient samples. In some cases, each patient sample from the j quantity of patient samples may include an multi omics profile of the constituents (e.g., proteins, RNA fragments, and / or the like) of a biopsy sample (e.g., liquid biopsy, tissue biopsy, and / or the like). Accordingly, in some cases, the j quantity of patient samples may be split, for example, randomly, to form the k quantity of holdout folds. In some cases, each holdout fold of the resulting k quantity of holdout folds may include a quantity of patient samples. In some cases, a single dataset may be split into multiple holdout sets, such as a first holdout set and a second holdout set, with the patient samples in the dataset being shuffled differently across the holdout folds such that the holdout folds in each holdout set exhibit a different distribution of patient samples. As described in more details below, each holdout fold may be combined with a training and testing set that excludes samples already included in the holdout fold. Each iteration of computational feature selection may include training and testing one or instances of the machine learning model on a training and testing set in order to identify a subset of features before different instances of the machine learning model trained using the same samples from the training and testing set to operate on the subset of features are tested using a holdout fold.

[0058] At 254, generate a plurality of training and testing sets by at least splitting, for each holdout fold in the one or more holdout sets, a remaining plurality of samples from the dataset that are not included in that holdout fold are split into a training and testing set. In some example embodiments, for a holdout fold from the k quantity of holdout folds in each holdout set,Attorney Ref.: 105892.228002 the patient samples in the dataset that are not included in the holdout fold may be further split into a k quantity of training and testing folds, such as an k — 1 quantity of training folds with a single testing fold. For example, the k quantity of holdout folds may include a first holdout fold and a second holdout fold. In some cases, the samples from the dataset that are not included in the first holdout fold may be split into a first training and testing set containing a k — 1 quantity of training folds and a single testing fold. Furthermore, the samples from the dataset that are not included in the second holdout fold may be split into a second training and testing set also containing a k — 1 quantity of training folds and a single testing fold.

[0059] At 256, a plurality of iterations of computational feature selection are performed using the one or more holdout sets and the plurality of training and testing sets. In some example embodiments, each iteration of computational feature selection may include training and testing a first instance of a machine learning model on a training and testing set to identify a subset of features. Moreover, in some cases, each iteration of computational feature selection may include a second instance of the machine learning model being trained using a same plurality of samples from the training and testing set to operate on the subset of features before the performance of the second instance of the machine learning model is tested on a holdout fold from the one or more holdout sets that excludes samples from the training and testing set. For clarity, the foregoing example includes one instance of the machine learning model being used to identify a subset of features while another instance of the machine learning model is used for testing on a holdout fold. However it should be appreciated that each iteration of computational feature selection may apply multiple instances of the machine learning model to identify the subset of features before multiple instances of the machine learning model are used for testing on the holdout fold.Attorney Ref: 105892.228002

[0060] In some cases, multiple iterations of computational feature selection may be performed using at least one holdout set and the corresponding training and testing sets. For example, during a first iteration of computational feature selection, a first instance of the machine learning model may be trained and tested on a first training and testing set. In some cases, a first subset of features that were selected with the highest frequency during the training and testing may be used to evaluate the performance of the machine learning model. For instance, in some cases, the first iteration of the computational feature selection may further include a second instance of the machine learning model being trained on the samples in the first training and testing set to operate on the first subset of features. Moreover, in some cases, the second instance of the machine learning model, upon being trained to operate on the first subset of features, may be applied to operate on a first holdout fold, which excludes samples from the first training and testing set. As described in more details below, the performance of the second instance of the machine learning model operating on the first holdout fold may provide an estimate of the performance of the machine learning model applied to real world data.

[0061] During a second iteration of computational feature selection, a third instance of the machine learning model may be trained and tested on a second training and testing set. In some cases, a second subset of features that were selected with the highest frequency during the training and testing may be used to evaluate the performance of the machine learning model. For example, in some cases, the second iteration of the computational feature selection may further include a fourth instance of the machine learning model being trained on the samples in the second training and testing set to operate on the second subset of features. Moreover, in some cases, the fourth instance of the machine learning model, upon being trained to operate on the second subsetAttorney Ref: 105892.228002 of features, may be applied to operate on a second holdout fold that excludes samples from the second training and testing set.

[0062] In some example embodiments, the features included in the first subset of features from the first iteration of computational feature selection and the second subset of features from the second iteration of computational feature selection may be selected based on the frequency with which they are selected by the machine learning model operating on the corresponding training and testing folds. For example, in instances where the machine learning model is a regularized regression model (e.g., a logistic or linear regression model with regularization such as LI regularization, L2 regularization, elastic net, and / or the like), the features in each subset of features may be selected based on the coefficients that are learned by the regularized logistic regression model. In some cases, regularization of machine learning model (e.g., the logistic regression model, the linear regression model, the support vector machine (SVM), and / or the like_ may include reducing (e.g., to a zero value) the values of coefficients that fail to satisfy one or more thresholds (e.g., minimum value). For example, in some cases, regularization of the machine learning model may be achieved with the addition of a penalty term in the corresponding log likelihood function. Doing so may minimize the contribution of the corresponding features (e g., genes) to the classification (e.g., probability distribution across the set of possible classifications) output by the logistic regression model. Moreover, the use of regularization (e.g., LI regularization, L2 regularization, elastic net, and / or the like) may give rise to features (e.g., genes) with non-zero value coefficients and features (e.g., genes) with zero value coefficients. In the context of the various implementations of computational feature selection described herein, a feature (e.g., gene) may be considered “selected,” for example, as a relevant or important feature the feature is assigned a non-zero value coefficient by the regularized logisticAttorney Ref: 105892.228002 regression model. Moreover, the number of times the feature (e.g., gene) is selected over multiple training and testing folds may be determined such that a feature that is selected a threshold quantity of times may be added to the subset of features. For instance, where the dataset exclusive of a holdout fold is further split into 10 training and testing folds (e.g., 9 training folds and one testing fold), the features included in the subset of features may include any feature that is selected for more than a threshold quantity of folds (e.g., 4 folds) during the training and validation. In some cases, for each iteration of computational feature selection, an additional instance of the machine learning model with the corresponding subset of features may be trained on the training and testing folds (e.g., 9 training folds and one testing fold) to determine the coefficients of each feature included in the subset of features. In some cases, this instance of the machine learning model, which has coefficients set for the subset of features identified through multiple iterations of computational feature selection, may then undergo testing on a corresponding holdout fold.

[0063] As described in more details below, the performance of the machine learning model as applied to real world data may be determined based on the performance of the individual instances of the machine learning model generated at each iteration of computational feature selection, each of which being trained to operate on a different subset of features. For instance, in the foregoing example, in addition to the performance of the second instance of the machine learning model (e.g., from the first iteration of computational feature selection) operating on the first holdout set, the performance of the machine learning model applied to real world data may be further estimated based on the performance of the fourth instance of the machine learning model (e.g., from the second iteration of computational feature selection) operating on the second holdout fold. In instances where the holdout set includes an k quantity of holdout folds, an k quantity of iterations of computational feature selection may be performed. Moreover, whereAttorney Ref: 105892.228002 multiple holdout sets are generated from the dataset, the k quantity of iterations of computational feature selection may be repeated for each additional holdout set with a k quantity of holdout folds.

[0064] At 258, an aggregate subset of features present in the dataset is selected based at least on a frequency with which each feature in the dataset is selected across the plurality of iterations of computational feature selection. In some example embodiments, the aggregate subset of features, such as an n quantity of genes, may be selected based on the frequency with which each feature is selected across the plurality of iterations of computational feature selection performed, for example, using at least one holdout set with a k quantity of holdout folds and the corresponding training and testing sets. This aggregate subset of features, which is selected after multiple iterations of computational feature selection, is differentiated from the subsets of features used during each individual iteration of computational feature selection to evaluate the performance of the machine learning model. For example, in some cases, the machine learning model may be a regression model, such as a logistic regression model, a gradient boosted model (e.g., XGBoost), a support vector classifier (SVC), and / or the like, that estimates the probability of different classifications (e.g., disease or healthy, benign or malignant, disease subtype, disease stage, responder or non-responder, treatment selection, and / or the like) for a patient sample by learning optimal coefficients for combining the features included in the patient sample to generate a probability distribution across the set of different possible classifications. Where the patient sample includes a multiomics profile, each feature in the patient sample may correspond to a gene encoding a protein or an RNA fragment present in a biopsy specimen (e.g., liquid biopsy, tissue biopsy, and / or the like). Accordingly, in instances where the machine learning model is a logistic regression model, the machine learning model may be trained to learn, for each gene included in the multiomics profile, an optimal coefficient such that the classification (e.g., probabilityAttorney Ref: 105892.228002 distribution across the set of possible classifications) output by the machine learning model is consistent with the ground-truth classification for the multiomics profile.

[0065] In some example embodiments, the subset of features, such as the n quantity of genes, may be selected based on the coefficients that are learned by a regularized logistic regression model. As noted, in some cases, regularization (e.g., LI regularization) of the machine learning model (e.g., the logistic regression model, the linear regression model, the support vector machine (SVM), and / or the like) may include reducing (e.g., to a zero value) the values of coefficients that fail to satisfy one or more thresholds (e.g., minimum value), for example, by the addition of a penalty term in the corresponding log likelihood function, thereby minimizing the contribution of the corresponding features (e.g., genes) to the classification (e.g., probability distribution across the set of possible classifications) output by the machine learning model. The use of regularization may give rise to features (e.g., genes) with non-zero value coefficients and features (e.g., genes) with zero value coefficients, with the features assigned non-zero value coefficients being considered “selected” as relevant or important. Accordingly, the number of times the feature (e.g., gene) is selected over multiple iterations of computational feature selection may be determined such that the aggregate subset of features include those features that are selected a threshold quantity of times over multiple iterations of computational feature selection. For example, in instances where the dataset is split and shuffled to form 10 holdout sets, 10 holdout folds for each holdout set, and 10 training and testing folds for each holdout fold, each feature (e.g., gene) may be selected (e.g., by being assigned a non-zero coefficient value) an x number of times by the regularized machine learning model (e.g., logistic regression model, linear regression model, and / or support vector machine (SVM) with regularization such as LI regularization, L2 regularization, elastic net, and / or the like) operating each of the 1,000 different folds of the dataset.Attorney Ref: 105892.228002 Where the dataset includes an m quantity of genes, the n quantity of genes selected for inclusion in the subset of features may be an n quantity of the most frequently selected genes.

[0066] At 260, the performance of a machine learning model operating on the subset of features is estimated based at least on the performance of each instance of the machine learning model operating on a corresponding testing fold. In some example embodiments, the performance of a machine learning model operating on a subset of features from the dataset, such as an area under the curve (AUC) confidence interval indicative of the uncertainty in the output of the machine learning model, may be determined based on the performance of the different instances of the machine learning model operating on the k quantity of holdout folds in each holdout set, the corresponding k quantity of testing folds, and / or the like. For example, in some cases, the performance of the machine learning model operating on the n quantity of genes to determine a classification for one or more patient samples may correspond to a range of performance (e.g., range of AUCs) achieved by the machine learning model operating on the k quantity of holdout folds in each holdout set. Alternatively and / or additionally, the performance of the machine learning model operating on the n quantity of genes to determine a classification for one or more patient samples may correspond to a mean (or average) performance (e.g., AUC) achieved by the machine learning model operating on the testing fold associated with each of the k quantity of holdout folds across the one or more holdout sets. In some cases, instances of the machine learning model whose performance (e.g., AUC) fail to satisfy certain thresholds (e.g., AUC < 0.5 ) may be excluded.

[0067] To further illustrate the splitting of a dataset into training folds, testing folds, and holdout folds, FIG. 3 depicts a schematic diagram illustrating the dataset 135 being split into training, testing, and holdout folds across multiple iterations of computational feature selection.Attorney Ref: 105892.228002 Referring to FIGS. 1, 2A-B, and 3, in some cases, the feature selection engine 110 may split the dataset 135 into different training folds, testing folds, and holdout folds across multiple iterations of computational feature section. For example, in some cases, the feature selection engine 110 may split the dataset into a k quantity of holdout folds for Holdout Set 1. In some cases, the feature selection engine 110 may shuffle the samples in the dataset 135 to in order to generate a different split of the dataset 135 to form another k quantity of holdout folds for up to a k quantity of holdout sets (e.g., up to Holdout Set k).

[0068] Referring again to FIG. 3, for each holdout fold from the k quantity of holdout folds in a holdout set (e g., from Holdout Set 1 to Holdout Set fc), the feature selection engine 110 may generate a training and testing set from the samples not included in the holdout fold. For example, FIG. 3 shows on possible split in which the samples not included in the holdout fold are split into a k quantity of folds, including a k-1 quantity of training folds (e g., Training Folds 1 through fc-1) and a single Testing Fold (e.g., Testing Fold / c). As shown in FIG. 3, Iteration 1 of computational feature selection includes a first machine learning model instance 305a being trained on Training Folds k — 1 before being tested on Testing Fold k and Holdout Fold 1. For example, in some cases, the first machine learning model instance 305a may be trained on Training Folds k — 1 and tested on Testing Fold k. A first subset of features that were most frequently selected during the training and testing may be used to evaluate the performance of the first machine learning model instance 305a. For instance, in some cases, a second machine learning model instance 305b may be trained on the samples in the Training Folds k — 1 and the Testing Fold k to operate on the first subset of features. Once the second machine learning model instance 305b is trained to operate on the first subset of features, the second machine learning model instance 305b may be applied to Holdout Fold 1 in order to determine the performance of theAttorney Ref: 105892.228002 machine learning model 125 operating on the first subset of features. It should be appreciated that, in some cases, the first machine learning model 305a and the second machine learning model instance 305b may be different instances of the machine learning model 125. While the first machine learning model instance 305a and the second machine learning model instance 305b may share the same architecture, the first machine learning model instance 305a and the second machine learning model instance 305b are nevertheless different models that operate on different features and applies different coefficients.

[0069] In some cases, Iteration 2 of computation feature selection may include a third machine learning model instance 305c being trained on Training Folds k-1 before being tested on Testing Fold k and Holdout Fold 2. For example, in some cases, the third machine learning model instance 305c may be trained on Training Folds k — 1 and tested on Testing Fold k. In some cases, the third machine learning model instance 305c may be yet another instance of the machine learning model 125 that shares the same architecture as the first machine learning model instance 305a and the second machine learning model instance 305b but operates on different features and applies different coefficients.

[0070] In some cases, a second subset of features that were most frequently selected during the training and testing may be used to evaluate the performance of the machine learning model 125. For example, in some cases, a fourth machine learning model instance 305d may be trained on the samples in the Training Folds k — 1 and the Testing Fold k to operate on the second subset of features. Once the fourth machine learning model instance 305d is trained to operate on the second subset of features, the fourth machine learning model instance 305d of the machine learning model 125 may be applied to Holdout Fold 2 in order to determine the performance of the machine learning model 125 operating on the second subset of features. In some cases, theAttorney Ref: 105892.228002 aforementioned training and testing may be repeated for a k quantity of iterations, with Iteration k of computational feature selection training and validating another instance of the machine learning model 125 on Training Folds k — 1 and Testing Fold k before the performance of the machine learning model 125 operating on a subset of features is evaluated using Holdout Fold k.[00071J As noted, the dataset 135 may be split into multiple holdout sets, with multiple iterations of computational feature selection being performed for the holdout folds in each holdout set. For example, FIG. 3 shows the dataset 135 being split into a k quantity of holdout sets, with each holdout set including a k quantity of holdout folds. Accordingly, FIG. 3 shows a single iteration of computational feature selection being performed for each holdout fold such that a k X k quantity of iterations of computational feature selection are performed for the k quantity of holdout folds in each holdout set in the k quantity of holdout sets. In some example embodiments, an aggregate subset of features from the dataset 135 that are most frequently selected across the k X k quantity of iterations of computational feature selection may be identified for use by the machine learning model 125 in deployment. For instance, in the example shown in FIG. 3, different instances of the machine learning model 125, each having the same architecture but different features and coefficients, may be applied to operate on a k x k x k quantity of training, testing, and holdout folds generated from the dataset 135. As a regularized machine learning model (e.g., a logistic regression model, a linear regression model, and / or a support vector machine (SVM) with regularization such as LI regularization, L2 regularization, elastic net, and / or the like), the machine learning model 125 may “select” a subset of features (e.g., genes) for each fold by at least assigning non-zero coefficient values to some but not all of the features present in the dataset 135.Attorney Ref: 105892.228002

[0072] In some example embodiments, the number of folds for which a feature (e.g., gene) from the dataset 135 is assigned a non-zero coefficient value may be aggregated over the k x k x k quantity of training, testing, and holdout folds. In some cases, those features that were most frequently assigned a non-zero coefficient value, and are thus selected as relevant or important for the most number of training, testing, and holdout folds, may be identified for inclusion in the aggregate subset of features. Moreover, in some cases, the aggregate subset of features may be applied in one or more downstream tasks. For example, in some cases, the machine learning model 125 that is deployed to operate on real life data may be configured to operate on the aggregate subset of features, meaning that the machine learning model 125 may perform a cognitive task (e.g., classification task, generative task, and / or the like) based on the aggregate subset of features. Alternatively and / or additionally, in instances where the features in the aggregate subset of features are biomarkers (e.g., genes encoding one or more proteins, RNA fragments, and / or the like), the aggregate subset of features may be used in a targeted assay panel.

[0073] In some example embodiments, the performance of each instance of the machine learning model 125 on a testing fold or a holdout fold may be quantified by one or more performance metrics including, for example, accuracy (overall correctness of classifications), precision (correctness of positive predictions), recall or true positive rate (ability to capture all positive instances), Fl score (a combination of recall and precision), false positive rate, area under the receiver operating characteristic (ROC) curve (AUC), and / or the like. Moreover, in some cases, the performance of the machine learning model 125 deployed to operate on real life data (e.g., multiomics profiles of real life patient biopsy specimen) may be estimated based on the performance of the machine learning model 125 operating on the testing folds and holdout folds across the k X k x k iterations of computational feature selection. For example, in some cases,Attorney Ref: 105892.228002 the values of a performance metric (e.g., AUC) for the machine learning model 125 operating on the testing fold across all k X k x k iterations of computational feature selection may be used to determine a mean (or average) performance metric. Alternatively and / or additionally, the values of the performance metric (e.g., AUC) for the machine learning model 125 operating on holdout folds across all k x k x k iterations of computational feature selection may be used to determine a range or distribution of the values for the performance metric.

[0074] Experimental Examples

[0075] In some example embodiments, a logistic regression model with LI regularization underwent 30 repeats of being trained and tested on 10 training and testing folds (e.g., 10-fold cross validation) before being evaluated with a holdout fold from a holdout set containing 10 holdout folds. The training, testing, and holdout folds were generated by repeated splitting and shuffling of 94 samples and 2778 mapped genes from the Early Detection Non-Small Cell Lung Cancer (NSCLC) dataset. During the repeated iterations of computational feature selection, 108 genes were selected and the performance of the logistic regression model was evaluated by applying instances of the logistic regression model trained to operating on these 108 genes on the holdout folds.

[0076] As shown in the graph 400 depicted in FIG. 4, which illustrates the number of times each gene is selected, 19 out of the 108 genes were selected for more than 200 of the 1200 possible folds. Moreover, FIG. 4 shows that the top 4 genes were selected in more than 1000 of the 1200 possible folds. In this experimental example, the logistic regression model with LI regularization achieved a mean (or average) AUC of 0.815 with a 95% confidence interval (0.80, 0.83). FIG. 5 depicts a graph 800 illustrating the distribution of distribution of AUC scores for the holdout folds as well as the average training fold AUC scores for each holdout fold. The AUCAttorney Ref.: 105892.228002 scores shown in FIG. 5 were achieved when the logistic regression model considered genes that were selected in at least 9 out of the 10 training and testing folds.

[0077] Varying the required number of times a gene is selected by the logistic regression model in order to be included in the aggregate subset of features is shown to have some impact on the performance of the logistic regression model. Graph 600 in FIG. 6 shows the distribution of distribution of AUC scores for the holdout folds as well as the average training fold AUC scores for each holdout fold when the logistic regression model operated on genes that were selected in at least 7 out of the 10 training and testing folds. Graph 700 in FIG. 7 shows the distribution of AUC scores for the holdout folds as well as the average training fold AUC scores for each holdout fold when the logistic regression model operated on genes that were selected in at least 8 out of the 10 training and testing folds. In those examples, the regularized logistic regression model performed similarly on the holdout folds and performed slightly better on the testing folds, which indicates a possible increase in overfitting towards the training set when more genes are selected. In order words, a higher threshold, which decreases the number of genes that qualify for inclusion in the aggregate subset of features, tend to improve the performance of the logistic regression model.

[0078] FIG. 8 depicts a block diagram illustrating an example of computing system 800, in accordance with some example embodiments. Referring to FIGS. 1 and 8, the computing system 800 may be used to implement the feature selection engine 110, the machine learning engine 120, the data store 130, the client device 140, and / or any components therein.

[0079] As shown in FIG. 8, the computing system 800 can include a processor 810, a memory 820, a storage device 830, and input / output devices 840. The processor 810, the memory 820, the storage device 830, and the input / output devices 840 can be interconnected via a systemAttorney Ref.: 105892.228002 bus 850. The processor 810 is capable of processing instructions for execution within the computing system 800. Such executed instructions can implement one or more components of, for example, the feature selection engine 110, the machine learning engine 120, the data store 130, the client device 140, and / or the like. In some example embodiments, the processor 810 can be a single-threaded processor. Alternately, the processor 810 can be a multi -threaded processor. The processor 810 is capable of processing instructions stored in the memory 820 and / or on the storage device 830 to display graphical information for a user interface provided via the input / output device 840.

[0080] The memory 820 is a computer readable medium such as volatile or nonvolatile that stores information within the computing system 800. The memory 820 can store data structures representing configuration object databases, for example. The storage device 830 is capable of providing persistent storage for the computing system 800. The storage device 830 can be a floppy disk device, a hard disk device, an optical disk device, or a tape device, or other suitable persistent storage means. The input / output device 840 provides input / output operations for the computing system 800. In some example embodiments, the input / output device 840 includes a keyboard and / or pointing device. In various implementations, the input / output device 840 includes a display unit for displaying graphical user interfaces.

[0081] According to some example embodiments, the input / output device 840 can provide input / output operations for a network device. For example, the input / output device 840 can include Ethernet ports or other networking ports to communicate with one or more wired and / or wireless networks (e.g., a local area network (LAN), a wide area network (WAN), the Internet).Attorney Ref: 105892.228002

[0082] In some example embodiments, the computing system 800 can be used to execute various interactive computer software applications that can be used for organization, analysis and / or storage of data in various formats. Alternatively, the computing system 800 can be used to execute any type of software applications. These applications can be used to perform various functionalities, e.g., planning functionalities (e.g., generating, managing, editing of spreadsheet documents, word processing documents, and / or any other objects, etc.), computing functionalities, communications functionalities, etc. The applications can include various add-in functionalities or can be standalone computing products and / or functionalities. Upon activation within the applications, the functionalities can be used to generate the user interface provided via the input / output device 840. The user interface can be generated and presented to a user by the computing system 800 (e.g., on a computer screen monitor, etc.).

[0083] One or more aspects or features of the subject matter described herein can be realized in digital electronic circuitry, integrated circuitry, specially designed ASICs, field programmable gate arrays (FPGAs) computer hardware, firmware, software, and / or combinations thereof. These various aspects or features can include implementation in one or more computer programs that are executable and / or interpretable on a programmable system including at least one programmable processor, which can be special or general purpose, coupled to receive data and instructions from, and to transmit data and instructions to, a storage system, at least one input device, and at least one output device. The programmable system or computing system may include clients and servers. A client and server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.Attorney Ref: 105892.228002

[0084] These computer programs, which can also be referred to as programs, software, software applications, applications, components, or code, include machine instructions for a programmable processor, and can be implemented in a high-level procedural and / or object-oriented programming language, and / or in assembly / machine language. As used herein, the term “machine-readable medium” refers to any computer program product, apparatus and / or device, such as for example magnetic discs, optical disks, memory, and Programmable Logic Devices (PLDs), used to provide machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term “machine-readable signal” refers to any signal used to provide machine instructions and / or data to a programmable processor. The machine-readable medium can store such machine instructions non-transitorily, such as for example as would a non-transient solid-state memory or a magnetic hard drive or any equivalent storage medium. The machine-readable medium can alternatively or additionally store such machine instructions in a transient manner, such as for example, as would a processor cache or other random access memory associated with one or more physical processor cores.

[0085] To provide for interaction with a user, one or more aspects or features of the subject matter described herein can be implemented on a computer having a display device, such as for example a cathode ray tube (CRT) or a liquid crystal display (LCD) or a light emitting diode (LED) monitor for displaying information to the user and a keyboard and a pointing device, such as for example a mouse or a trackball, by which the user may provide input to the computer. Other kinds of devices can be used to provide for interaction with a user as well. For example, feedback provided to the user can be any form of sensory feedback, such as for example visual feedback, auditory feedback, or tactile feedback; and input from the user may be received in anyAttorney Ref: 105892.228002 form, including acoustic, speech, or tactile input. Other possible input devices include touch screens or other touch-sensitive devices such as single or multi-point resistive or capacitive track pads, voice recognition hardware and software, optical scanners, optical pointers, digital image capture devices and associated interpretation software, and the like.

[0086] In the descriptions above and in the claims, phrases such as “at least one of’ or “one or more of’ may occur followed by a conjunctive list of elements or features. The term “and / or” may also occur in a list of two or more elements or features. Unless otherwise implicitly or explicitly contradicted by the context in which it used, such a phrase is intended to mean any of the listed elements or features individually or any of the recited elements or features in combination with any of the other recited elements or features. For example, the phrases “at least one of A and B;” “one or more of A and B;” and “A and / or B” are each intended to mean “A alone, B alone, or A and B together.” A similar interpretation is also intended for lists including three or more items. For example, the phrases “at least one of A, B, and C;” “one or more of A, B, and C;” and “A, B, and / or C” are each intended to mean “A alone, B alone, C alone, A and B together, A and C together, B and C together, or A and B and C together.” Use of the term “based on,” above and in the claims is intended to mean, “based at least in part on,” such that an unrecited feature or element is also permissible.

[0087] The subject matter described herein can be embodied in systems, apparatus, methods, and / or articles depending on the desired configuration. The implementations set forth in the foregoing description do not represent all implementations consistent with the subject matter described herein. Instead, they are merely some examples consistent with aspects related to the described subject matter. Although a few variations have been described in detail above, other modifications or additions are possible. In particular, further features and / or variations can beAttorney Ref: 105892.228002 provided in addition to those set forth herein. For example, the implementations described above can be directed to various combinations and subcombinations of the disclosed features and / or combinations and subcombinations of several further features disclosed above. In addition, the logic flows depicted in the accompanying figures and / or described herein do not necessarily require the particular order shown, or sequential order, to achieve desirable results. Other implementations may be within the scope of the following claims.

Claims

Attorney Ref.: 105892.228002CLAIMSWhat is claimed is:

1. A system, comprising:at least one data processor; andat least one memory storing instructions, which when executed by the at least one data processor, result in operations comprising:receiving a dataset including a plurality of samples, wherein each sample in the plurality of samples include a set of features;splitting the dataset into one or more holdout sets;generating a plurality of training and testing sets by at least splitting, for each holdout fold in the one or more holdout sets, a remaining plurality of samples not included in the holdout fold into a training and testing set;performing a plurality of iterations of computational feature selection using the one or more holdout sets and the plurality of training and testing sets, wherein each iteration of computational feature selection includes training and validating a first instance of a machine learning model on a training and testing set, wherein each iteration of computational feature selection further includes training, using a same plurality of samples in the training and testing set, a second instance of the machine learning model to operate on a subset of features selected during the training and testing of the first instance of the machine learning model before applying the second instance of the machine learning model to a holdout fold from the one or more holdout sets;Attorney Ref.: 105892.228002 selecting, based at least on a frequency with which each feature present in the dataset is selected across the plurality of iterations of computational feature selection, an aggregate subset of features from the dataset; andapplying the aggregate subset of features to one or more downstream tasks.

2. The system of claim 1, wherein each iteration of computational feature selection includestraining the first instance of the machine learning model on a plurality of training folds in the training and testing set, andvalidating the first instance of the machine learning model on a testing fold in the training and testing set.

3. The system of any of claims 1 to 2, wherein the plurality of iterations of computational feature selection includes a first iteration in which the first instance of the machine learning model is trained and tested on a first training and testing set before the second instance of the machine learning model applied to a first holdout fold from a first holdout set.

4. The system of claim 3, wherein the plurality of iterations of computational feature selection includes a second iteration in whicha third instance of the machine learning model is trained and tested on a second training and testing set, anda fourth instance of the machine learning model is trained using the same plurality of samples in the second training and testing set to operate on a second subset of data selected during the training and testing of the third instance of the machine learning model.Attorney Ref: 105892.228002 5. The system of claim 4, wherein the second iteration further includes applying the fourth instance of the machine learning model to a second holdout fold from the first holdout set.

6. The system of any of claims 4 to 5, wherein the second iteration further includes applying the fourth instance of the machine learning model to a second holdout fold from a second holdout set.

7. The system of any of claims 1 to 6, wherein the operations further comprise: determining, based at least on a performance of the machine learning model applied to a testing fold in each training and testing set, an average performance of the machine learning model.

8. The system of any of claims 1 to 7, wherein the operations further comprise: determining, based at least on a performance of the machine learning model applied to each holdout fold, a distribution of the performance of the machine learning model.

9. The system of any of claims 1 to 8, wherein the machine learning model assigns each feature of the plurality of features a coefficient for combining the plurality of features into an output of the machine learning model, and wherein the logistic regression model includes regularization to reduce, to a zero value, any coefficient whose value fails to satisfy one or more thresholds.

10. The system of claim 9, wherein a feature is selected when a coefficient associated with the feature is assigned a non-zero value by the regularization.

11. The system of any of claims 9 to 10, wherein the regularization comprises LI regularization, L2 regularization, and / or elastic net.Attorney Ref: 105892.228002 12. The system of any of claims 1 to 11, wherein the one or more downstream tasks include a new instance of the machine learning model trained to generate an output by at least operating on the aggregate subset of features.

13. The system of claim 12, wherein the output includes a classification of a patient sample to indicate one or more of disease or healthy, benign or malignant, disease subtype, disease stage, responder or non-responder, or treatment selection.

14. The system of any of claims 11 to 13, wherein the one or more downstream tasks includes an assay panel targeting a plurality of genes comprising the aggregate subset of features.

15. The system of any of claims 1 to 14, wherein the plurality of samples include a plurality of patient samples.

16. The system of any of claims 1 to 15, wherein the plurality of samples include a plurality of multiomics profdes.

17. The system of any of claims 1 to 16, wherein the set of features include one or more genes expressing a protein or a ribonucleic acid (RNA) fragment present in a biopsy specimen.

18. The system of any of claims 1 to 17, wherein a first plurality of iterations of computational feature selection is performed for a first holdout set, and wherein a second plurality of iterations of computational feature selection is performed for a second holdout set.

19. The system of claim 18, wherein the first holdout set and the second holdout set each includes a plurality of holdout folds, and wherein each holdout fold is associated with aAttorney Ref: 105892.228002 plurality of training folds and a testing fold generated by splitting the remaining plurality of samples.

20. The system of any of claims 1 to 19, wherein the machine learning model comprises one or more of a logistic regression model, a linear regression model, or a support vector machine.

21. A computer-implemented method, comprising:receiving a dataset including a plurality of samples, wherein each sample in the plurality of samples include a set of features;splitting the dataset into one or more holdout sets;generating a plurality of training and testing sets by at least splitting, for each holdout fold in the one or more holdout sets, a remaining plurality of samples not included in the holdout fold into a training and testing set;performing a plurality of iterations of computational feature selection using the one or more holdout sets and the plurality of training and testing sets, wherein each iteration of computational feature selection includes training and validating a first instance of a machine learning model on a training and testing set, wherein each iteration of computational feature selection further includes training, using a same plurality of samples in the training and testing set, a second instance of the machine learning model to operate on a subset of features selected during the training and validating of the first instance of the machine learning model before applying the second instance of the machine learning model to a holdout fold from the one or more holdout sets;Attorney Ref: 105892.228002 selecting, based at least on a frequency with which each feature present in the dataset is selected across the plurality of iterations of computational feature selection, an aggregate subset of features from the dataset; andapplying the aggregate subset of features to one or more downstream tasks.

22. A non-transitory computer readable medium storing instructions, which when executed by at least one data processor, result in operations comprising:receiving a dataset including a plurality of samples, wherein each sample in the plurality of samples include a set of features;splitting the dataset into one or more holdout sets;generating a plurality of training and testing sets by at least splitting, for each holdout fold in the one or more holdout sets, a remaining plurality of samples not included in the holdout fold into a training and testing set;performing a plurality of iterations of computational feature selection using the one or more holdout sets and the plurality of training and testing sets, wherein each iteration of computational feature selection includes training and validating a first instance of a machine learning model on a training and testing set, wherein each iteration of computational feature selection further includes training, using a same plurality of samples in the training and testing set, a second instance of the machine learning model to operate on a subset of features selected during the training and validating of the first instance of the machine learning model before applying the second instance of the machine learning model to a holdout fold from the one or more holdout sets;Attorney Ref.: 105892.228002 selecting, based at least on a frequency with which each feature present in the dataset is selected across the plurality of iterations of computational feature selection, an aggregate subset of features from the dataset; andapplying the aggregate subset of features to one or more downstream tasks..

Images