Systems and methods for deep-and-wide machine learning

Abstract

Systems and methods for deep-and-wide learning (DWL) in accordance with embodiments of the invention are illustrated. One embodiment includes a system for training a machine learning model, including a processor, and a memory, the memory containing a DWL application that configures the processor to obtain training data includes a plurality of data samples, extract high-dimensional features from the training data, wherein the high-dimensional features capture intra-sample characteristics of individual data samples, extract low-dimensional features from the training data using dimensionality reduction wherein the low-dimensional features capture inter-sample relationships among the plurality of data samples, integrate the high-dimensional features and the low-dimensional features to form a combined feature vector, and train a machine learning model using the combined feature vector.

Description

Systems and Methods for Deep-and-Wide Machine LearningSTATEMENT OF FEDERALLY SPONSORED RESEARCH

[0001] This invention was made with Government support under contracts CA256890, CA275772, and LM014309 awarded by the National Institutes of Health. The Government has certain rights in the invention.CROSS-REFERENCE TO RELATED APPLICATIONS

[0002] This application claims priority to U. S. Application No. 63 / 750,241, titled Sample Interrelationship-Informed Representation Learning for High-Performance Data-Driven Inference, filed January 27, 2025, which is hereby incorporated by reference in its entirety.FIELD OF INVENTION

[0003] The present disclosure relates to machine learning and neural network architectures, and more particularly to a learning framework that integrates highdimensional intra-sample features with low-dimensional inter-sample relationship features for enhanced data-driven inference and model training efficiency and accuracy.BACKGROUND

[0004] Neural networks are computational models inspired by biological neural systems that process information through layers of interconnected nodes. These networks may receive input data, transform the data through weighted connections and activation functions across multiple layers, and produce output predictions or classifications. Neural networks have been applied to various domains including image recognition, natural language processing, and predictive analytics. In supervised machine learning, models are trained using labeled data where input-output pairs guide the learning process to map inputs to desired outputs. In unsupervised machine learning, models identify patterns and structures within unlabeled data without predefined output targets, enabling discovery of hidden relationships and groupings within datasets.SUMMARY

[0005] Systems and methods for deep-and-wide learning (DWL) in accordance with embodiments of the invention are illustrated. One embodiment includes a system for training a machine learning model, including a processor, and a memory, the memory containing a DWL application that configures the processor to obtain training data includes a plurality of data samples, extract high-dimensional features from the training data, wherein the high-dimensional features capture intra-sample characteristics of individual data samples, extract low-dimensional features from the training data using dimensionality reduction wherein the low-dimensional features capture inter-sample relationships among the plurality of data samples, integrate the high-dimensional features and the low-dimensional features to form a combined feature vector, and train a machine learning model using the combined feature vector.

[0006] In a further embodiment, the dimensionality reduction is a Bayesian dimensionality reduction models which reduces data dimensionality by projecting latent variables through a projection matrix with Gaussian noise incorporated to account for variability in the observed data.

[0007] In still another embodiment, the DWL learning application further configures the processor to employ variational inference to approximate a posterior distribution of the latent variables and the projection matrix.

[0008] In a still further embodiment, the DWL learning application further configures the processor to apply automatic relevance determination priors to determine relevant latent dimensions.

[0009] In yet another embodiment, extracting high-dimensional features includes processing the training data through at least one deep learning layer.

[0010] In a yet further embodiment, the DWL learning application further configures the processor to process the combined feature vector through a learnable deep learning layer to produce an intermediate vector and apply an activation function to the intermediate vector.

[0011] In another additional embodiment, the machine learning model includes a transformer-based architecture.

[0012] One embodiment includes a method for training a machine learning model, including obtaining training data includes a plurality of data samples, extracting highdimensional features from the training data, wherein the high-dimensional features capture intra-sample characteristics of individual data samples, extracting lowdimensional features from the training data using dimensionality reduction, wherein the low-dimensional features capture inter-sample relationships among the plurality of data samples, integrating the high-dimensional features and the low-dimensional features to form a combined feature vector, and training a machine learning model using the combined feature vector.

[0013] In a further additional embodiment, the dimensionality reduction is Bayesian dimensionality reduction which models observed data as generated by projecting latent variables through a projection matrix with Gaussian noise incorporated to account for variability in the observed data.

[0014] In another embodiment again, the method further includes steps for employing variational inference to approximate a posterior distribution of the latent variables and the projection matrix.

[0015] In a further embodiment again, the method further includes steps for applying automatic relevance determination priors to determine relevant latent dimensions.

[0016] In still yet another embodiment, extracting high-dimensional features includes processing the training data through at least one deep learning layer convolutional and pooling layers.

[0017] In a still yet further embodiment, the method further includes steps for processing the combined feature vector through a learnable deep learning to produce an intermediate vector and applying an activation function to the intermediate vector.

[0018] In still another additional embodiment, the machine learning model includes a transformer-based architecture.

[0019] One embodiment includes a system including a processor, and a memory, the memory containing a trained machine learning model, wherein the trained machine learning model is trained by obtaining training data includes a plurality of data samples, extracting high-dimensional features from the training data, wherein the high-dimensional features capture intra-sample characteristics of individual data samples, extracting low-dimensional features from the training data using dimensionality reduction, wherein the low-dimensional features capture inter-sample relationships among the plurality of data samples, integrating the high-dimensional features and the low-dimensional features to form a combined feature vector, and training the machine learning model using the combined feature vector.

[0020] In a still further additional embodiment, the dimensionality reduction is Bayesian dimensionality reduction which models input data as generated by projecting latent variables through a projection matrix with Gaussian noise incorporated to account for variability in the input data.

[0021] In still another embodiment again, training the machine learning model includes employing variational inference to approximate a posterior distribution of the latent variables and the projection matrix.

[0022] In a still further embodiment again, training the machine learning model includes applying automatic relevance determination priors to determine relevant latent dimensions.

[0023] In yet another additional embodiment, extracting high-dimensional features includes processing the training data through at least one deep learning layer.

[0024] In a yet further additional embodiment, training the machine learning model includes processing the combined feature vector through a learnable deep learning layer to produce an intermediate vector and applying an activation function to the intermediate vector. The foregoing general description of the illustrative embodiments and the following detailed description thereof are merely exemplary aspects of the teachings of this disclosure and are not restrictive.

[0025] Additional embodiments and features are set forth in part in the description that follows, and in part will become apparent to those skilled in the art upon examination of the specification or may be learned by the practice of the invention. A further understanding of the nature and advantages of the present invention may be realized by reference to the remaining portions of the specification and the drawings, which forms a part of this disclosure.BRIEF DESCRIPTION OF FIGURES

[0026] FIG. 1 illustrates a DWL learning system in accordance with an embodiment of the invention.

[0027] FIG. 2 illustrates a block diagram of a DWL learning server in accordance with an embodiment of the invention.

[0028] FIG. 3 illustrates a flowchart for a DWL training process for a machine learning model in accordance with an embodiment of the invention.

[0029] FIG. 4 illustrates a block diagram of a DWL learning architecture in accordance with an embodiment of the invention.DETAILED DESCRIPTION

[0030] Data-driven inference using deep neural networks (DNNs) involves extracting feature representations from input data samples to perform tasks such as classification and generation. Generally, training a DNN is to find the optimal nonlinear function that maps an input data manifold to an output manifold, defined either by the labels for supervised learning or by other criteria for unsupervised learning. At the core of this mapping process is the creation of vector representations of input data samples through automatic feature extraction across multiple layers in a neural network. Since input data are typically interrelated, it is imperative to preserve their relationships throughout the learning process and the construction of feature representations of the samples. However, this is rarely the case in current deep learning paradigms, where feature extraction is performed sample-by-sample without explicitly accounting for their overall relationships. This sample interrelationship-agnostic approach significantly hampers the data-driven learning process, slowing down training and resulting in suboptimal solutions.

[0031] In order to address these issues, a Deep-and-Wide Learning (DWL) framework is described which captures the underlying structure of the input dataset in a set of low dimensional (LD) feature vectors. The underlying structure is used to guide highdimensional (HD) feature extraction across different layers. By leveraging the interactions between LD and HD features, the sample interrelationship-informed DWL framework facilitates the rapid generation of more meaningful data representations and significantlyenhances pattern discovery. Unlike conventional DNNs which focus solely on HD feature extraction from individual inputs in a sample-independent fashion with multiple layers, DWL extracts both LD and HD feature representations from the dataset and uses them synergistically to achieve significantly improved learning.

[0032] In numerous embodiments, a Bayesian approach is used to capture the overall relationships among data samples by first learning the distribution of the input samples in the input manifold space, and then leveraging the distribution to produce an LD representation that preserves structural integrity between input manifold and LD space. Simultaneously, the HD representation can be generated using feature extraction techniques to capture the characteristic details of individual inputs. The HD and LD features can be aligned through a dense layer to ensure consistency of the learned HD and LD representations. With the integration of LD features, DWL achieves superior learning performance, with significantly faster training convergence even when using only a single block of feature extraction. Experimentally, DWL has not only outperformed current DNNs in inference accuracy by 6-45%, but also achieved orders of magnitude of improvement in computational efficiency (5x to 200x).

[0033] The DWL framework may be applied as a front-end component to any arbitrary machine learning model to improve performance of the machine learning model. The framework operates by extracting both HD and LD features from input data and providing the combined features to a downstream machine learning model for inference tasks. This modular approach allows the DWL learning framework to augment existing deep neural network architectures without requiring modification of the underlying model structure.

[0034] In many embodiments, the DWL framework achieves 15- to 200-fold reduction in training time compared to conventional deep neural networks. Empirically, the DWL framework has achieved 6% to 45% improvement in downstream task accuracy compared to conventional models. These performance improvements result from the synergistic interaction between HD features capturing individual sample properties and LD features capturing global data structure relationships.

[0035] A significant advantage of the DWL framework is that almost any machine learning model can be enhanced simply by utilizing DWL to modify its inputs. That is, the efficiency and performance gains can be achieved across many different models,generally improving the field of machine learning and artificial intelligence (AI). As AI models become more integrated, the limitation of physical compute capacity has become an increasingly significant issue. DWL can be used to increase the productivity of existing compute capacity, providing significant benefits not only to large data centers, but enabling deployment of Al models on lower-capability machines where previously not computationally viable. DWL systems are described below, followed by a discussion of the DWL framework and methods for use.DWL Systems

[0036] Turning now to FIG. 1, a DWL system in accordance with an embodiment of the invention is illustrated. DWL system 100 includes a DWL server 110, a server cluster 120, a client device 130, and a network 140 that interconnects these components. The DWL server 110 may be configured to execute DWL operations and implement the DWL framework for training machine learning models. The server cluster 120 may provide distributed computing resources that can be utilized for parallel processing of training data, feature extraction operations, or inference tasks. In some embodiments, the server cluster may operate in conjunction with the DWL server to handle computationally intensive workloads or to scale processing capacity based on demand. In various embodiments, the server cluster stores training data for use in training machine learning models using the DWL framework. The client device 130 may be configured to interact with the DWL server or the server cluster to submit data, initiate learning tasks, configure model parameters, receive inference results, or perform any other interaction as appropriate to the requirements of specific applications of embodiments of the invention.

[0037] The network 140 serves as the communication infrastructure that facilitates data exchange between the DWL learning server 110, the server cluster 120, and the client device 130. In various embodiments, the network 140 may include local area networks, wide area networks, the Internet, or combinations thereof. The topology of the DWL learning system 100 allows flexible communication pathways between components, enabling distributed processing and remote access to DWL capabilities. As can be readily appreciated, other computing architectures can be used to implement the DWL framework as described below.

[0038] Turning now to FIG. 2, a block diagram for a DWL server in accordance with an embodiment of the invention is illustrated. DWL server 200 includes a processor 210. The processor can be any logic processing circuit including (but not limited to) a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), and / or any other logic circuit capable of carrying out DWL processes as described herein. DWL server 200 further includes an input / output (I / O) interface 220.The I / O interface 220 facilitates communication between the DWL server 200 and external devices or networks, enabling data exchange with client devices, server clusters, and other computing resources. The DWL server 200 also includes a memory 230 that stores data and instructions utilized by the server. The memory 230 may include volatile memory such as random access memory (RAM), non-volatile memory such as solid-state drives, hard disk drives, or other persistent storage media, or any combination thereof. Stored within the memory 230 is a deep learning application 232 that provides machine-readable instructions for configuring the processor to execute DWL processes. The memory 230 further contains a DeepWide learning model 234 that implements the DWL framework for extracting both HD features capturing intrinsic properties of individual data samples and LD features characterizing the global structure and inter-sample relationships within datasets. The processor 210 is operatively connected to the I / O interface 220 and the memory 230, enabling the processor 210 to access and execute the deep learning application 232 and the DeepWide learning model 234, and to communicate with external systems through the I / O interface 220. As can be readily appreciated, other computing architectures, such as distributed systems, can be used without departing from the scope or spirit of the invention. DWL processes are discussed in further detail below.DWL Processes

[0039] DWL processes are those that implement the DWL framework. A significant advantage of the DWL framework is that any arbitrary machine learning model can be enhanced by preprocessing inputs using DWL. In numerous embodiments, this preprocessing is achieved using a neural network that may be integrated into the arbitrary machine learning model, or as a separate model that passes its input to a separatearbitrary machine learning model. In either case, the preprocessing model and the arbitrary machine learning model together are referred to herein as a DWL model, as the arbitrary machine learning model utilizes the preprocessing model to generate its input.

[0040] Turning now to FIG. 3, a DWL process for training an arbitrary machine learning model in accordance with an embodiment of the invention is illustrated. Process 300 begins with obtaining (310) training data. The training data can be any arbitrary set of training data of arbitrary type appropriate for training the desired machine learning model for a desired purpose. The training data is then put through two separate transformations. HD features are extracted (320), and separately a LD feature representation is extracted (330). In numerous embodiments, HD features are extracted using traditional deep learning methods. For example, a CNN would use convolution, max pooling, and activations to extract HD features. Generalized, this can be formalized as, for an input X, the extracted intra-sample features can be written as Fh = fh(X ; θh), where fh represents the sequence of operations, θh denotes the parameters of these operations, and Fh is the resulting feature set derived from X.

[0041] LD feature representations can be extracted in a number of ways as well. LD feature extraction captures inter-sample relationships and preserves the manifold structure of input data. In numerous embodiments, Bayesian dimensionality reduction (BDR) is employed to extract LD features. The extracted LD features can be expressed as Fi = fi(X ; θl), where fi represents the operations involved in Bayesian manifold learning, θl denotes the parameters of the Bayesian approach, and Fi is the resulting LD feature set derived from input X. BDR is a probabilistic framework that reduces the dimensionality of a dataset while preserving intrinsic structure and patterns within the data.

[0042] The BDR approach projects input data (X) onto a LD latent space (Z) while preserving meaningful relationships between data points. This process is modeled as a probabilistic generative process expressed as:p(X, Z, Q) = p(Z)p(Q)p(X|Z, Q)

[0043] where X is the input data, Z represents the latent variables forming the LD embedding, Q is a projection matrix that maps latent variables to the observed space, and p(Z), p(Q), and p(X|Z, Q) denote prior distributions and a likelihood function,respectively. This formulation allows the model to infer posterior distributions of the latent variables p(Z|X) and the projection matrix p(Q|X) based on observed data.

[0044] In various embodiments, the observed data is assumed to be generated by projecting the latent variables through the projection matrix, with noise incorporated to account for variability in the data. This relationship can be expressed as:X = QZ + E

[0045] where E represents Gaussian noise. The goal of BDR is to infer the most probable latent variables (Z) and projection matrix (Q) that explain the observed data. This inference can be performed through the posterior distribution:p(Z, Q|X) ∝ p(X|Z, Q)p(Z)p(Q).

[0046] Given the complexity of Bayesian models, the posterior distribution p(Z, Q|X) is often intractable. To address this computational challenge, variational inference may be employed as an approximate inference method. Variational inference approximates the true posterior p(Z, Q|X) with a simpler, tractable distribution q(Z, Q), which is factorized as:q(Z, Q) = q(Z)q(Q)This factorization enables efficient computation by reducing dependencies between variables. The parameters of q(Z) and q(Q) are optimized to minimize the divergence from the true posterior.

[0047] In order to prevent overfitting and to enhance model generalization, prior distributions can be incorporated to regularize the model. In various embodiments, element-wise priors can be employed here each element of the projection matrix Q is assigned an independent prior. For example, a gamma distribution prior on the precision parameter c|>,ypromotes sparsity in the matrix:p(φi,j) = Gamma(αφ , βφ) This encourages individual entries of Q to be close to zero unless strongly supported by the data.

[0048] In many embodiments, automatic relevance determination (ARD) priors can be employed, where priors are shared across dimensions to automatically determine the most relevant latent dimensions. The ARD prior on cp7(shared across the j-th column of Q) is defined as p(φj) = Gamma(αφ , βφ)

[0049] This prior encourages the model to focus on the most informative dimensions, effectively performing feature selection. The posterior distribution can be modeled over the latent variables (q(Z)) as a Gaussian distribution. This enables computation of the mean and covariance of the latent variables, providing a compact and probabilistic representation of the data in the reduced-dimensional space. The projection matrix Q can be inferred as a Gaussian distribution as well, with its mean representing the most likely projection and its variance capturing the uncertainty in the projection. Once posterior distributions have been inferred, the data can be projected into the reduced-dimensional space using the mean of the orthogonalized projection matrix: u = XTQorth, where Qorth is derived from the posterior mean of Q.

[0050] ARD provides a number of advantages over existing methods. First, ARD provides uncertainty quantification. Probabilistic outputs provide confidence estimates for the latent representations and projections, enabling better uncertainty management. ARD is also highly flexible. The use of priors allows the model to incorporate domain-specific knowledge, making it adaptable to diverse datasets. ARD improves robustness as well, as the probabilistic framework is more resilient to noise and outliers, ensuring stable performance under varying conditions. ARD, as noted, enables automatic feature selection which focuses the model on the most relevant dimensions, simplifying the analysis and improving efficiency. Finally, ARD enhances interpretability. The latent space and projections become more interpretable through the incorporation of priors and probabilistic assumptions, aiding in model transparency. However, as noted above, the use of ARD is not a requirement of DWL, and other prior methods can be used as appropriate to the requirements of specific applications of embodiments of the invention.

[0051] In alternative embodiments, LD feature extraction may be achieved using classical machine learning methods instead of the Bayesian approach. Such alternative methods include Principal Component Analysis (PCA), Multidimensional Scaling (MDS), Linear Discriminant Analysis (LDA), and Independent Component Analysis (ICA). While these classical approaches may enhance learning when integrated into the DWL framework, empirical results indicate that the Bayesian approach may provide superior performance compared to these alternatives in many applications.

[0052] Since intra- and inter-sample features originate from different manifold domains, additional steps can be taken to align them for final decision making. Once the HD and LD features are extracted, they are aligned (340) together into a single feature vector using concatenation: F = concat(Fh, Fl). In some embodiments, other methods of aggregation (for example, summing) can be used in place of concatenation. The aggregated feature vector F can then process through a dense layer, producing an intermediate vector \ / . V is subsequently passed through an activation function (e.g., ReLU). This sequence of concatenation, projection, and activation during forward and backward propagation ensures that the features from two distinct data spaces are aligned within a shared feature space.

[0053] Depending on the specific task (classification or regression), the final output vector can be derived from V. For classification tasks with k classes, the output is computed as P = Softmax(Wc■ V + be), where Wcand bcare the weights and biases of the classification layer, and P represents the probability distribution across the classes. For regression tasks, the output is given by Y = Wr· V + br, where Wrand brare the weights and biases of the regression layer, and Y represents the predicted output values. This concatenated vector is then provided as to train (350) the arbitrary machine learning model. This process can occur for each element of training data in the original training data set. Similarly, once trained, the inputs can be passed through the same preprocessing steps during operation.

[0054] Referring now to FIG. 4, a graphical depiction DWL architecture in accordance with an embodiment of the invention is illustrated. The DWL architecture receives input data and processes the input data through two parallel pathways that extract features from distinct representational spaces. The top channel uses BDR followed by a fully connected layer (FCL), a rectified linear unit (ReLU) layer, a second FCL, and a second ReLU layer to extract low dimensional features. The bottom channel extracts high dimensional features using a convolution layer, a pooling layer, a dropout layer, a ReLU layer, followed by a FCL layer. The outputs of the top and bottom channels are aligned into a feature vector and provided to the machine learning model.

[0055] In various embodiments, the DWL learning model may operate in a single-channel framework where LD representations are computed after HD features are extracted at selected layers. In the single-channel framework, the input data is first processed through the HD feature extraction pathway, and at one or more selected intermediate layers, the extracted HD features are used as input to compute corresponding LD representations. The LD representations computed at the selected layers are then combined with the HD features to ensure that HD and LD features remain consistent throughout the network during training. This single-channel implementation provides similar performance characteristics to the dual-channel implementation while offering an alternative architectural configuration that may be suited to particular application requirements.Example DWL Applications

[0056] DWL can be applied, as noted above, to any arbitrary model. Several examples are provided below demonstrating the breadth of possibility for DWL application. The DWL framework may be applied to foundation models for processing genomic data. By way of example, a DWL foundation model (DWL-FM) can be built on a BERT architecture for processing single-cell RNA sequencing (scRNA-seq) data. The DWL-FM architecture includes six transformer encoder units followed by a dense layer for processing the genomic data. The transformer encoder units process input sequences through self-attention mechanisms that capture relationships between genes within each cell's transcriptome.

[0057] In the DWL-FM, LD features are projected through a dense layer onto a 32-dimensional space. The 32-dimensional LD representation is then concatenated with HD features extracted by the transformer encoder units. This concatenation aligns the intersample relationships captured by the LD features with the intra-sample characteristics captured by the HD features from the transformer encoders.

[0058] The DWL-FM employs a rank-based encoding scheme for representing gene expression data. In the rank-based encoding, genes are ranked by their expression level within each cell. The rank values are then normalized against gene expression across the entire dataset used for training. This rank-based encoding provides a nonparametric representation of each cell's transcriptome that prioritizes genes that are informative fordistinguishing cell states. Highly expressed housekeeping genes are de-emphasized in the ranking, while transcription factors or other state-specific genes, even when expressed at low levels, are assigned higher ranks based on their relative expression within the cell. The rank-based encoding also enhances robustness to technical artifacts, as the relative ranking of genes within a cell remains stable even when absolute transcript counts may vary due to experimental conditions.

[0059] During pretraining of the DWL-FM, a masked language modeling objective can be employed. In the masked language modeling approach, 15% of the genes in each transcriptome are masked, and the model is trained to predict the masked genes based on the unmasked context. This self-supervised pretraining approach leverages large-scale unlabeled data independent of sample labels. The masked language modeling objective enables the DWL-FM to learn gene network dynamics and hierarchical relationships within the attention weights through self-supervised learning. Following pretraining, the DWL-FM may be fine-tuned on various downstream tasks to improve predictive accuracy for specific applications such as cell type annotation, disease classification, or other genomic analysis tasks.

[0060] In another example, the DWL framework may be applied to GPT2 architecture for text classification tasks. In various embodiments, a DWL-GPT2 model integrates the dual-channel feature extraction approach with a pretrained GPT2 model. The GPT2 model provides HD feature extraction through its transformer-based architecture, while the LD feature extraction channel captures inter-sample relationships within the text dataset. The LD features and HD features extracted by GPT2 are concatenated and processed through a classification layer for text categorization.

[0061] In some embodiments, the DWL-GPT2 model processes text datasets containing factory reports with categorical labels. The factory reports include plain text descriptions and associated categorical labels that define classification targets. The text data may be split into training, validation, and testing sets for model development and evaluation. During training, the pretrained GPT2 model may be frozen while the classification layers and LD feature extraction components are fine-tuned. The integration of LD features with the GPT2 embeddings enables the model to leverage both thecontextual representations learned by GPT2 and the global inter-sample relationships captured by the LD feature extraction channel.

[0062] By way of further example, the DWL framework may be applied to convolutional neural networks for image classification tasks. In various embodiments, the DWL framework is integrated with AlexNet architecture. The AlexNet architecture provides HD feature extraction through convolutional layers, pooling layers, and fully connected layers. The LD feature extraction channel operates in parallel to capture intersample relationships within the image dataset. The HD features from AlexNet and the LD features are concatenated to form a combined feature representation for classification.

[0063] In some embodiments, the DWL framework is integrated with VGG16 architecture. The VGG16 architecture employs a deep stack of convolutional layers with small receptive fields for HD feature extraction. The LD feature extraction channel captures global data structure relationships that complement the detailed spatial features extracted by VGG16. The concatenated HD and LD features are processed through classification layers for image categorization tasks.

[0064] In various embodiments, the DWL framework is integrated with VGG19 architecture. The VGG19 architecture extends the VGG16 architecture with additional convolutional layers for deeper HD feature extraction. The integration of LD features with VGG19 enables the model to leverage both the hierarchical spatial representations and the inter-sample relationships within the dataset.

[0065] In various applications, the DWL framework is integrated with ResNet architecture. ResNet employs residual connections that enable training of deep networks for HD feature extraction. The residual connections allow gradient flow through the network during training. The LD features captured by the Bayesian dimensionality reduction approach are concatenated with the ResNet features to form a combined representation for classification tasks.

[0066] The DWL framework may be integrated with Xception architecture in some embodiments. Xception employs depthwise separable convolutions for efficient HD feature extraction. The LD feature extraction channel operates in parallel to capture global data structure, and the resulting LD features are concatenated with the Xception features for downstream classification.

[0067] In various embodiments, the DWL framework is integrated with GoogleNet architecture. GoogleNet employs inception modules that process input at multiple scales for HD feature extraction. The multi-scale features from GoogleNet are combined with LD features that capture inter-sample relationships to form a comprehensive feature representation for classification.

[0068] The DWL framework may be integrated with DenseNet architecture in some embodiments. DenseNet employs dense connections where each layer receives feature maps from all preceding layers for HD feature extraction. The dense connectivity pattern enables feature reuse throughout the network. The LD features extracted through Bayesian dimensionality reduction are concatenated with the DenseNet features to leverage both intra-sample characteristics and inter-sample relationships for image classification tasks.

[0069] DWL-enhanced image classification has further been used to great success on radiology images for disease detection and diagnosis. Additional improvements to different technical fields, let alone machine learning generally, are contemplated as a result of the above.

[0070] A number of implementations have been described. Nevertheless, it will be understood that various modifications may be made without departing from the spirit and scope of the disclosure. Accordingly, other implementations are within the scope of the following claims.

Claims

CLAIMS1. A system for training a machine learning model, comprising:a processor; anda memory, the memory containing a deep-and-wide learning (DWL) application that configures the processor to:obtain training data comprising a plurality of data samples;extract high-dimensional features from the training data, wherein the highdimensional features capture intra-sample characteristics of individual data samples;extract low-dimensional features from the training data using dimensionality reduction wherein the low-dimensional features capture intersample relationships among the plurality of data samples;integrate the high-dimensional features and the low-dimensional features to form a combined feature vector; andtrain a machine learning model using the combined feature vector.

2. The system of claim 1, wherein the dimensionality reduction is a Bayesian dimensionality reduction which reduces data dimensionality by projecting latent variables through a projection matrix with Gaussian noise incorporated to account for variability in the observed data.

3. The system of claim 2, wherein the DWL learning application further configures the processor to employ variational inference to approximate a posterior distribution of the latent variables and the projection matrix.

4. The system of claim 3, wherein the DWL learning application further configures the processor to apply automatic relevance determination priors to determine relevant latent dimensions.

5. The system of claim 1, wherein extracting high-dimensional features comprises processing the training data through at least one deep learning layer.

6. The system of claim 1, wherein the DWL learning application further configures the processor to process the combined feature vector through a learnable deep learning layer to produce an intermediate vector and apply an activation function to the intermediate vector.

7. The system of claim 1, wherein the machine learning model comprises a transformer-based architecture.

8. A method for training a machine learning model, comprising:obtaining training data comprising a plurality of data samples; extracting high-dimensional features from the training data, wherein the high-dimensional features capture intra-sample characteristics of individual data samples;extracting low-dimensional features from the training data using dimensionality reduction, wherein the low-dimensional features capture intersample relationships among the plurality of data samples;integrating the high-dimensional features and the low-dimensional features to form a combined feature vector; andtraining a machine learning model using the combined feature vector.

9. The method of claim 8, wherein the dimensionality reduction is a Bayesian dimensionality reduction which models observed data as generated by projecting latent variables through a projection matrix with Gaussian noise incorporated to account for variability in the observed data.

10. The method of claim 9, further comprising employing variational inference to approximate a posterior distribution of the latent variables and the projection matrix.

11. The method of claim 10, further comprising applying automatic relevance determination priors to determine relevant latent dimensions.

12. The method of claim 8, wherein extracting high-dimensional features comprises processing the training data through at least one deep learning layer convolutional and pooling layers.

13. The method of claim 8, further comprising processing the combined feature vector through a learnable deep learning to produce an intermediate vector and applying an activation function to the intermediate vector.

14. The method of claim 8, wherein the machine learning model comprises a transformer-based architecture.

15. A system comprising:a processor; anda memory, the memory containing a trained machine learning model, wherein the trained machine learning model is trained by:obtaining training data comprising a plurality of data samples; extracting high-dimensional features from the training data, wherein the high-dimensional features capture intra-sample characteristics of individual data samples;extracting low-dimensional features from the training data using dimensionality reduction, wherein the low-dimensional features capture intersample relationships among the plurality of data samples;integrating the high-dimensional features and the low-dimensional features to form a combined feature vector; andtraining the machine learning model using the combined feature vector.

16. The system of claim 15, wherein the dimensionality reduction is a Bayesian dimensionality reduction which models input data as generated by projecting latent variables through a projection matrix with Gaussian noise incorporated to account for variability in the input data.

17. The system of claim 16, wherein training the machine learning model comprises employing variational inference to approximate a posterior distribution of the latent variables and the projection matrix.

18. The system of claim 17, wherein training the machine learning model comprises applying automatic relevance determination priors to determine relevant latent dimensions.

19. The system of claim 15, wherein extracting high-dimensional features comprises processing the training data through at least one deep learning layer.

20. The system of claim 15, wherein training the machine learning model comprises processing the combined feature vector through a learnable deep learning layer to produce an intermediate vector and applying an activation function to the intermediate vector.

21. A system for training a machine learning model, comprising:a processor; anda memory, the memory containing a deep-and-wide learning (DWL) application that configures the processor to:obtain training data comprising a plurality of data samples;extract high-dimensional features from the training data, wherein the highdimensional features capture intra-sample characteristics of individual data samples;extract low-dimensional features from the training data using Bayesian dimensionality reduction, where the low-dimensional features capture intersample relationships among the plurality of data samples;integrate the high-dimensional features and the low-dimensional features to form a combined feature vector; andtrain a machine learning model using the combined feature vector.

22. A system comprising:a processor; anda memory, the memory containing a trained machine learning model, wherein the trained machine learning model is trained by:obtaining training data comprising a plurality of data samples; extracting high-dimensional features from the training data, wherein the high-dimensional features capture intra-sample characteristics of individual data samples;extracting low-dimensional features from the training data using Bayesian dimensionality reduction, where the low-dimensional features capture intersample relationships among the plurality of data samples;integrating the high-dimensional features and the low-dimensional features to form a combined feature vector; andtraining the machine learning model using the combined feature vector.

23. A method for training a machine learning model, comprising:obtaining training data comprising a plurality of data samples; extracting high-dimensional features from the training data, wherein the high-dimensional features capture intra-sample characteristics of individual data samples;extracting low-dimensional features from the training data using Bayesian dimensionality reduction, where the low-dimensional features capture intersample relationships among the plurality of data samples;integrating the high-dimensional features and the low-dimensional features to form a combined feature vector; andtraining a machine learning model using the combined feature vector.

Images