Systems and methods for training neural networks and related machine learning models

Abstract

Systems and methods for training-concurrent compression of network models are illustrated. One embodiment includes a method for training-concurrent pruning, performed by processors. The method obtains a pretrained network comprising initial values for a potential function and weights. The method trains a network based upon the pretrained network and training samples, wherein training the network comprises calculating loss for a current network using a loss function; updating a gradient of the potential function at the current time iteration according to a gradient of the loss; updating an inverse mapping of the gradient of the potential function; creating an updated network including updated weights for a next time iteration according to the gradient of the potential function and the inverse mapping of the gradient of the potential function, wherein the updated weights is obtained by pruning the weights of the current network; and storing the updated weights.

Description

Systems and Methods for Training Neural Networks and Related Machine Learning ModelsCROSS-REFERENCE TO RELATED APPLICATIONS

[0001] The current application claims the benefit of and priority under 35 U. S. C. § 119(e) to U. S. Provisional Patent Application No. 63 / 749,353, entitled “Systems and Methods for Training Neural Networks and Related Machine Learning Models,” filed January 24, 2025. The disclosure of U. S. Provisional Patent Application No. 63 / 749,353 is hereby incorporated by reference in its entirety for all purposes.FIELD OF THE INVENTION

[0002] The present invention generally relates to training neural networks and, more specifically, selecting and using loss and potential functions to train neural networks.BACKGROUND

[0003] Generative models are systems configured to produce text, images, videos, or other (e.g., media) products based on collections of gathered training data. These models are capable of using deep learning techniques to learn underlying patterns and structures from training data and using these patterns and structures to produce new products. As such, artificial intelligence (Al) systems based on the use of generative models have uses across a wide range of industries, including software development, entertainment, customer service, marketing, art, and design. Development of such systems typically depends on the application of machine learning (e.g., deep learning) techniques in a manner that focuses on the utilization of neural networks to perform tasks such as classification and representation.

[0004] Deep learning refers to using artificial neural networks as computational network models for learning representations from training data. These network models are trained utilizing example data points from training sets, and tuning their internal parameters (e.g., weights) so that the model’s predictions align well with the given (labels of the) data points. The most important aspect of this process is to learn representations that can be applied to unknown prompts / queries. In doing so, the effectiveness of network models can be measured by efficiency in predicting results based on unseen data points.SUMMARY OF THE INVENTION

[0005] Systems and methods for training-concurrent compression of neural network models are illustrated. One embodiment includes a method for trainingconcurrent pruning, performed by one or more processors. The method obtains a pretrained neural network model, wherein the pretrained neural network model comprises initial values for a potential function and a set of weights. The method trains a neural network model based upon the pretrained neural network model and a set of training samples, wherein training the neural network model comprises, for each of a plurality of time iterations: calculating loss for a current neural network model having a current set of weights at a current time iteration using a loss function; updating a gradient of the potential function at the current time iteration according to a gradient of the loss; updating an inverse mapping of the gradient of the potential function; creating an updated neural network model including an updated set of weights for a next time iteration according to the gradient of the potential function and the inverse mapping of the gradient of the potential function, wherein the updated set of weights is obtained by pruning at least a portion of the current set of weights of the current neural network model; and storing the updated set of weights.

[0006] In a further embodiment, the trained neural network model is a multi-layered neural network model.

[0007] In another embodiment, the multi-layered neural network model is selected from the group consisting of a convolutional neural network model, a recursive neural network model, a multi-layer perceptron, and a generative adversarial network.

[0008] In yet another embodiment, the multi-layered neural network model is a large language model.

[0009] In still another embodiment, the updated set of weights is further obtained based on at least one additional vector comprising a current set of auxiliary variables for the current neural network model.

[0010] In another embodiment, each sample of the set of training samples corresponds to a different auxiliary variable of the current set of auxiliary variables.

[0011] In still yet another embodiment, obtaining the updated set of weights for the next time iteration comprises obtaining an updated set of auxiliary variables based on the current set of weights for the current time iteration.

[0012] In another embodiment, each auxiliary variable of the current set of auxiliary variables is obtained based on a regularization parameter and the loss function evaluated for the current set of weights at the current time iteration.

[0013] In a further embodiment, a regularizer mirror descent (RMD) process is used to obtain at least one selected from the group consisting of the updated set of weights; and the current set of auxiliary variables.

[0014] In another embodiment, iterative updates are used to obtain at least one selected from the group consisting of the updated set of weights; and the updated set of auxiliary variables.

[0015] In still another embodiment, each of the current set of weights and the current set of auxiliary variables is initialized to at least one selected from the group consisting of: a zero vector and a non-zero vector comprising random numbers within a predetermined threshold of zero.

[0016] In yet another embodiment, the loss function comprises a constraintenforcing loss computed from the current set of auxiliary variables.

[0017] In a further embodiment, the constraint-enforcing loss is applied to prevent overfitting the neural network model to the set of training samples.

[0018] In another embodiment, a stochastic mirror descent (SMD) process is used to obtain the updated set of weights.

[0019] In still another embodiment, the loss function includes a Kurtosis term.

[0020] In yet another embodiment, the Kurtosis term is a function of the current set of weights.

[0021] In another embodiment, supervised learning is used to train the neural network model.

[0022] In a further embodiment, the loss function is configured to only equal zero when a predicted output value of the neural network model exactly matches a labeled output value from the set of training samples.

[0023] In still another embodiment, the loss function is selected from the group consisting of a quadratic loss function and a cross-entropy loss function.

[0024] In another embodiment, at the current time iteration, obtaining the updated set of weights is based, at least in part, on a learning rate at the current time iteration.

[0025] In yet another embodiment, the learning rate is time-invariant.

[0026] In a further embodiment, updating the gradient of the potential function at the current time iteration is performed using at least one system selected from the group consisting of Adam, AdamW, AdaGrad, RMSProp, Momentum, and WeightDecay.

[0027] In another embodiment, the gradient of the potential function at the current time iteration is obtained recursively based on the gradient of the potential function at a previous time iteration.

[0028] In still another embodiment, training the neural network model at the current time iteration further comprises storing the gradient of the potential function at the current time iteration.

[0029] In yet another embodiment, after the plurality of time iterations, the potential function is minimized via implicit regularization.

[0030] In another embodiment, the potential function is a strictly convex function.

[0031] In a further embodiment, the potential function is a norm function.

[0032] In still another embodiment, the potential function is selected from the group consisting of: an £qnorm function where q » 1; and an £1+enorm function where 1 » £ > 0.

[0033] In another embodiment, training the neural network model comprises at least one selected from the group consisting of distributed training, batch training, synchronous training, ensemble learning, meta learning, transfer learning, and continual learning.

[0034] In yet another embodiment, the inverse mapping of the gradient of the potential function is a many-to-one function.

[0035] In a further embodiment, obtaining the updated set of weights bimodally concentrates a distribution of the set of weights.

[0036] In another embodiment, quantizing the neural network model after the plurality of time iterations is performed according to a bimodality of the distribution of the set of weights.

[0037] In still another embodiment, a data size of each individual weight of the current set of weights is quantized to a single bit.

[0038] In yet another embodiment, the potential function is selected to induce sparsity within the set of weights.

[0039] In another embodiment, the potential function is selected to induce group sparsity within a trained set of weights for the trained neural network model.

[0040] In a further embodiment, the potential function is an elastic net.

[0041] One embodiment includes a method for training-concurrent quantization, performed by one or more processors. The method obtains a pretrained neural network model, wherein the pretrained neural network model comprises initial values for a potential function and a set of weights. The method trains a neural network model based upon the pretrained neural network model and a set of training samples, wherein training the neural network model comprises, for each of a plurality of time iterations: calculating loss for a current neural network model having a current set of weights at a current time iteration using a loss function; updating a gradient of the potential function at the current time iteration according to a gradient of the loss; updating an inverse mapping of the gradient of the potential function; creating an updated neural network model including an updated set of weights for a next time iteration according to the gradient of the potential function and the inverse mapping of the gradient of the potential function, wherein creating the updated neural network model bimodally concentrates a distribution of the updated set of weights; and storing the updated set of weights. The method quantizes the trained neural network model after the plurality of time iterations, wherein quantizing the trained neural network model is performed according to a bimodality of the distribution of a trained set of weights.

[0042] In a further embodiment, the trained neural network model is a multi-layered neural network model.

[0043] In another embodiment, the multi-layered neural network model is selected from the group consisting of a convolutional neural network model, a recursive neural network model, a multi-layer perceptron, and a generative adversarial network.

[0044] In yet another embodiment, the multi-layered neural network model is a large language model.

[0045] In still another embodiment, the updated set of weights is further obtained based on at least one additional vector comprising a current set of auxiliary variables for the current neural network model.

[0046] In another embodiment, each sample of the set of training samples corresponds to a different auxiliary variable of the current set of auxiliary variables.

[0047] In still yet another embodiment, obtaining the updated set of weights for the next time iteration comprises obtaining an updated set of auxiliary variables based on the current set of weights for the current time iteration.

[0048] In another embodiment, each auxiliary variable of the current set of auxiliary variables is obtained based on a regularization parameter and the loss function evaluated for the current set of weights at the current time iteration.

[0049] In a further embodiment, a regularizer mirror descent (RMD) process is used to obtain at least one selected from the group consisting of the updated set of weights; and the current set of auxiliary variables.

[0050] In another embodiment, iterative updates are used to obtain at least one selected from the group consisting of the updated set of weights; and the updated set of auxiliary variables.

[0051] In still another embodiment, each of the current set of weights and the current set of auxiliary variables is initialized to at least one selected from the group consisting of: a zero vector and a non-zero vector comprising random numbers within a predetermined threshold of zero.

[0052] In yet another embodiment, the loss function comprises a constraintenforcing loss computed from the current set of auxiliary variables.

[0053] In a further embodiment, the constraint-enforcing loss is applied to prevent overfitting the neural network model to the set of training samples.

[0054] In another embodiment, a stochastic mirror descent (SMD) process is used to obtain the updated set of weights.

[0055] In still another embodiment, the loss function includes a Kurtosis term.

[0056] In yet another embodiment, the Kurtosis term is a function of the current set of weights.

[0057] In another embodiment, supervised learning is used to train the neural network model.

[0058] In a further embodiment, the loss function is configured to only equal zero when a predicted output value of the neural network model exactly matches a labeled output value from the set of training samples.

[0059] In still another embodiment, the loss function is selected from the group consisting of a quadratic loss function and a cross-entropy loss function.

[0060] In another embodiment, at the current time iteration, obtaining the updated set of weights is based, at least in part, on a learning rate at the current time iteration.

[0061] In yet another embodiment, the learning rate is time-invariant.

[0062] In a further embodiment, updating the gradient of the potential function at the current time iteration is performed using at least one system selected from the group consisting of Adam, AdamW, AdaGrad, RMSProp, Momentum, and WeightDecay.

[0063] In another embodiment, the gradient of the potential function at the current time iteration is obtained recursively based on the gradient of the potential function at a previous time iteration.

[0064] In still another embodiment, training the neural network model at the current time iteration further comprises storing the gradient of the potential function at the current time iteration.

[0065] In yet another embodiment, after the plurality of time iterations, the potential function is minimized via implicit regularization.

[0066] In another embodiment, the potential function is a strictly convex function.

[0067] In a further embodiment, the potential function is a norm function.

[0068] In still another embodiment, the potential function is selected from the group consisting of: an £qnorm function where q » 1; and an £1+snorm function where 1 » E > 0.

[0069] In another embodiment, training the neural network model comprises at least one selected from the group consisting of distributed training, batch training, synchronous training, ensemble learning, meta learning, transfer learning, and continual learning.

[0070] In yet another embodiment, the inverse mapping of the gradient of the potential function is a many-to-one function.

[0071] In a further embodiment, a data size of each individual weight of the current set of weights is quantized to a single bit.

[0072] In another embodiment, the potential function is selected to enable the bimodality of the distribution of the trained set of weights.

[0073] In still another embodiment, the potential function is a rigid cage.

[0074] One embodiment includes a non-transitory machine-readable medium storing instructions that, when executed by a processor, are configured to cause the processor to perform operations for training-concurrent pruning. The processor obtains a pretrained neural network model, wherein the pretrained neural network model comprises initial values for a potential function and a set of weights. The processor trains a neural network model based upon the pretrained neural network model and a set of training samples, wherein training the neural network model comprises, for each of a plurality of time iterations: calculating loss for a current neural network model having a current set of weights at a current time iteration using a loss function; updating a gradient of the potential function at the current time iteration according to a gradient of the loss; updating an inverse mapping of the gradient of the potential function; creating an updated neural network model including an updated set of weights for a next time iteration according to the gradient of the potential function and the inverse mapping of the gradient of the potential function, wherein the updated set of weights is obtained by pruning at least a portion of the current set of weights of the current neural network model; and storing the updated set of weights.

[0075] In a further embodiment, the trained neural network model is a multi-layered neural network model.

[0076] In another embodiment, the multi-layered neural network model is selected from the group consisting of a convolutional neural network model, a recursive neural network model, a multi-layer perceptron, and a generative adversarial network.

[0077] In yet another embodiment, the multi-layered neural network model is a large language model.

[0078] In still another embodiment, the updated set of weights is further obtained based on at least one additional vector comprising a current set of auxiliary variables for the current neural network model.

[0079] In another embodiment, each sample of the set of training samples corresponds to a different auxiliary variable of the current set of auxiliary variables.

[0080] In still yet another embodiment, obtaining the updated set of weights for the next time iteration comprises obtaining an updated set of auxiliary variables based on the current set of weights for the current time iteration.

[0081] In another embodiment, each auxiliary variable of the current set of auxiliary variables is obtained based on a regularization parameter and the loss function evaluated for the current set of weights at the current time iteration.

[0082] In a further embodiment, a regularizer mirror descent (RMD) process is used to obtain at least one selected from the group consisting of the updated set of weights; and the current set of auxiliary variables.

[0083] In another embodiment, iterative updates are used to obtain at least one selected from the group consisting of the updated set of weights; and the updated set of auxiliary variables.

[0084] In still another embodiment, each of the current set of weights and the current set of auxiliary variables is initialized to at least one selected from the group consisting of: a zero vector and a non-zero vector comprising random numbers within a predetermined threshold of zero.

[0085] In yet another embodiment, the loss function comprises a constraintenforcing loss computed from the current set of auxiliary variables.

[0086] In a further embodiment, the constraint-enforcing loss is applied to prevent overfitting the neural network model to the set of training samples.

[0087] In another embodiment, a stochastic mirror descent (SMD) process is used to obtain the updated set of weights.

[0088] In still another embodiment, the loss function includes a Kurtosis term.

[0089] In yet another embodiment, the Kurtosis term is a function of the current set of weights.

[0090] In another embodiment, supervised learning is used to train the neural network model.

[0091] In a further embodiment, the loss function is configured to only equal zero when a predicted output value of the neural network model exactly matches a labeled output value from the set of training samples.

[0092] In still another embodiment, the loss function is selected from the group consisting of a quadratic loss function and a cross-entropy loss function.

[0093] In another embodiment, at the current time iteration, obtaining the updated set of weights is based, at least in part, on a learning rate at the current time iteration.

[0094] In yet another embodiment, the learning rate is time-invariant.

[0095] In a further embodiment, updating the gradient of the potential function at the current time iteration is performed using at least one system selected from the group consisting of Adam, AdamW, AdaGrad, RMSProp, Momentum, and WeightDecay.

[0096] In another embodiment, the gradient of the potential function at the current time iteration is obtained recursively based on the gradient of the potential function at a previous time iteration.

[0097] In still another embodiment, training the neural network model at the current time iteration further comprises storing the gradient of the potential function at the current time iteration.

[0098] In yet another embodiment, after the plurality of time iterations, the potential function is minimized via implicit regularization.

[0099] In another embodiment, the potential function is a strictly convex function.

[0100] In a further embodiment, the potential function is a norm function.

[0101] In still another embodiment, the potential function is selected from the group consisting of: an £qnorm function where q » 1; and an £1+snorm function where 1 » E > 0.

[0102] In another embodiment, training the neural network model comprises at least one selected from the group consisting of distributed training, batch training, synchronous training, ensemble learning, meta learning, transfer learning, and continual learning.

[0103] In yet another embodiment, the inverse mapping of the gradient of the potential function is a many-to-one function.

[0104] In a further embodiment, obtaining the updated set of weights bimodally concentrates a distribution of the set of weights.

[0105] In another embodiment, quantizing the neural network model after the plurality of time iterations is performed according to a bimodality of the distribution of the set of weights.

[0106] In still another embodiment, a data size of each individual weight of the current set of weights is quantized to a single bit.

[0107] In yet another embodiment, the potential function is selected to induce sparsity within the set of weights.

[0108] In another embodiment, the potential function is selected to induce group sparsity within a trained set of weights for the trained neural network model.

[0109] In a further embodiment, the potential function is an elastic net.

[0110] One embodiment includes a non-transitory machine-readable medium storing instructions that, when executed by a processor, are configured to cause the processor to perform operations for training-concurrent quantization. The processor obtains a pretrained neural network model, wherein the pretrained neural network model comprises initial values for a potential function and a set of weights. The processor trains a neural network model based upon the pretrained neural network model and a set of training samples, wherein training the neural network model comprises, for each of a plurality of time iterations: calculating loss for a current neural network model having a current set of weights at a current time iteration using a loss function; updating a gradient of the potential function at the current time iteration according to a gradient of the loss; updating an inverse mapping of the gradient of the potential function; creating an updated neural network model including an updated set of weights for a next time iteration according to the gradient of the potential function and the inverse mapping of the gradientof the potential function, wherein creating the updated neural network model bimodally concentrates a distribution of the updated set of weights; and storing the updated set of weights. The processor quantizes the trained neural network model after the plurality of time iterations, wherein quantizing the trained neural network model is performed according to a bimodality of the distribution of a trained set of weights.

[0111] In a further embodiment, the trained neural network model is a multi-layered neural network model.

[0112] In another embodiment, the multi-layered neural network model is selected from the group consisting of a convolutional neural network model, a recursive neural network model, a multi-layer perceptron, and a generative adversarial network.

[0113] In yet another embodiment, the multi-layered neural network model is a large language model.

[0114] In still another embodiment, the updated set of weights is further obtained based on at least one additional vector comprising a current set of auxiliary variables for the current neural network model.

[0115] In another embodiment, each sample of the set of training samples corresponds to a different auxiliary variable of the current set of auxiliary variables.

[0116] In still yet another embodiment, obtaining the updated set of weights for the next time iteration comprises obtaining an updated set of auxiliary variables based on the current set of weights for the current time iteration.

[0117] In another embodiment, each auxiliary variable of the current set of auxiliary variables is obtained based on a regularization parameter and the loss function evaluated for the current set of weights at the current time iteration.

[0118] In a further embodiment, a regularizer mirror descent (RMD) process is used to obtain at least one selected from the group consisting of the updated set of weights; and the current set of auxiliary variables.

[0119] In another embodiment, iterative updates are used to obtain at least one selected from the group consisting of the updated set of weights; and the updated set of auxiliary variables.

[0120] In still another embodiment, each of the current set of weights and the current set of auxiliary variables is initialized to at least one selected from the groupconsisting of: a zero vector and a non-zero vector comprising random numbers within a predetermined threshold of zero.

[0121] In yet another embodiment, the loss function comprises a constraintenforcing loss computed from the current set of auxiliary variables.

[0122] In a further embodiment, the constraint-enforcing loss is applied to prevent overfitting the neural network model to the set of training samples.

[0123] In another embodiment, a stochastic mirror descent (SMD) process is used to obtain the updated set of weights.

[0124] In still another embodiment, the loss function includes a Kurtosis term.

[0125] In yet another embodiment, the Kurtosis term is a function of the current set of weights.

[0126] In another embodiment, supervised learning is used to train the neural network model.

[0127] In a further embodiment, the loss function is configured to only equal zero when a predicted output value of the neural network model exactly matches a labeled output value from the set of training samples.

[0128] In still another embodiment, the loss function is selected from the group consisting of a quadratic loss function and a cross-entropy loss function.

[0129] In another embodiment, at the current time iteration, obtaining the updated set of weights is based, at least in part, on a learning rate at the current time iteration.

[0130] In yet another embodiment, the learning rate is time-invariant.

[0131] In a further embodiment, updating the gradient of the potential function at the current time iteration is performed using at least one system selected from the group consisting of Adam, AdamW, AdaGrad, RMSProp, Momentum, and WeightDecay.

[0132] In another embodiment, the gradient of the potential function at the current time iteration is obtained recursively based on the gradient of the potential function at a previous time iteration.

[0133] In still another embodiment, training the neural network model at the current time iteration further comprises storing the gradient of the potential function at the current time iteration.

[0134] In yet another embodiment, after the plurality of time iterations, the potential function is minimized via implicit regularization.

[0135] In another embodiment, the potential function is a strictly convex function.

[0136] In a further embodiment, the potential function is a norm function.

[0137] In still another embodiment, the potential function is selected from the group consisting of: an £qnorm function where q » 1; and an £1+enorm function where 1 >> £ > 0.

[0138] In another embodiment, training the neural network model comprises at least one selected from the group consisting of distributed training, batch training, synchronous training, ensemble learning, meta learning, transfer learning, and continual learning.

[0139] In yet another embodiment, the inverse mapping of the gradient of the potential function is a many-to-one function.

[0140] In a further embodiment, a data size of each individual weight of the current set of weights is quantized to a single bit.

[0141] In another embodiment, the potential function is selected to enable the bimodality of the distribution of the trained set of weights.

[0142] In still another embodiment, the potential function is a rigid cage.

[0143] One embodiment includes a computing system for training-concurrent pruning. The computing system includes a network interface; a memory storing instructions; and one or more processors configured to communicate with the network interface and the memory, the one or more processors further configured to execute the instructions to perform operations. The operations include obtaining a pretrained neural network model, wherein the pretrained neural network model comprises initial values for a potential function and a set of weights. The operations include training a neural network model based upon the pretrained neural network model and a set of training samples, wherein training the neural network model comprises, for each of a plurality of time iterations: calculating loss for a current neural network model having a current set of weights at a current time iteration using a loss function; updating a gradient of the potential function at the current time iteration according to a gradient of the loss; updating an inverse mapping of the gradient of the potential function; creating an updated neuralnetwork model including an updated set of weights for a next time iteration according to the gradient of the potential function and the inverse mapping of the gradient of the potential function, wherein the updated set of weights is obtained by pruning at least a portion of the current set of weights of the current neural network model; and storing the updated set of weights.

[0144] In a further embodiment, the trained neural network model is a multi-layered neural network model.

[0145] In another embodiment, the multi-layered neural network model is selected from the group consisting of a convolutional neural network model, a recursive neural network model, a multi-layer perceptron, and a generative adversarial network.

[0146] In yet another embodiment, the multi-layered neural network model is a large language model.

[0147] In still another embodiment, the updated set of weights is further obtained based on at least one additional vector comprising a current set of auxiliary variables for the current neural network model.

[0148] In another embodiment, each sample of the set of training samples corresponds to a different auxiliary variable of the current set of auxiliary variables.

[0149] In still yet another embodiment, obtaining the updated set of weights for the next time iteration comprises obtaining an updated set of auxiliary variables based on the current set of weights for the current time iteration.

[0150] In another embodiment, each auxiliary variable of the current set of auxiliary variables is obtained based on a regularization parameter and the loss function evaluated for the current set of weights at the current time iteration.

[0151] In a further embodiment, a regularizer mirror descent (RMD) process is used to obtain at least one selected from the group consisting of the updated set of weights; and the current set of auxiliary variables.

[0152] In another embodiment, iterative updates are used to obtain at least one selected from the group consisting of the updated set of weights; and the updated set of auxiliary variables.

[0153] In still another embodiment, each of the current set of weights and the current set of auxiliary variables is initialized to at least one selected from the groupconsisting of: a zero vector and a non-zero vector comprising random numbers within a predetermined threshold of zero.

[0154] In yet another embodiment, the loss function comprises a constraintenforcing loss computed from the current set of auxiliary variables.

[0155] In a further embodiment, the constraint-enforcing loss is applied to prevent overfitting the neural network model to the set of training samples.

[0156] In another embodiment, a stochastic mirror descent (SMD) process is used to obtain the updated set of weights.

[0157] In still another embodiment, the loss function includes a Kurtosis term.

[0158] In yet another embodiment, the Kurtosis term is a function of the current set of weights.

[0159] In another embodiment, supervised learning is used to train the neural network model.

[0160] In a further embodiment, the loss function is configured to only equal zero when a predicted output value of the neural network model exactly matches a labeled output value from the set of training samples.

[0161] In still another embodiment, the loss function is selected from the group consisting of a quadratic loss function and a cross-entropy loss function.

[0162] In another embodiment, at the current time iteration, obtaining the updated set of weights is based, at least in part, on a learning rate at the current time iteration.

[0163] In yet another embodiment, the learning rate is time-invariant.

[0164] In a further embodiment, updating the gradient of the potential function at the current time iteration is performed using at least one system selected from the group consisting of Adam, AdamW, AdaGrad, RMSProp, Momentum, and WeightDecay.

[0165] In another embodiment, the gradient of the potential function at the current time iteration is obtained recursively based on the gradient of the potential function at a previous time iteration.

[0166] In still another embodiment, training the neural network model at the current time iteration further comprises storing the gradient of the potential function at the current time iteration.

[0167] In yet another embodiment, after the plurality of time iterations, the potential function is minimized via implicit regularization.

[0168] In another embodiment, the potential function is a strictly convex function.

[0169] In a further embodiment, the potential function is a norm function.

[0170] In still another embodiment, the potential function is selected from the group consisting of: an £qnorm function where q » 1; and an £1+enorm function where 1 >> £ > 0.

[0171] In another embodiment, training the neural network model comprises at least one selected from the group consisting of distributed training, batch training, synchronous training, ensemble learning, meta learning, transfer learning, and continual learning.

[0172] In yet another embodiment, the inverse mapping of the gradient of the potential function is a many-to-one function.

[0173] In a further embodiment, obtaining the updated set of weights bimodally concentrates a distribution of the set of weights.

[0174] In another embodiment, quantizing the neural network model after the plurality of time iterations is performed according to a bimodality of the distribution of the set of weights.

[0175] In still another embodiment, a data size of each individual weight of the current set of weights is quantized to a single bit.

[0176] In yet another embodiment, the potential function is selected to induce sparsity within the set of weights.

[0177] In another embodiment, the potential function is selected to induce group sparsity within a trained set of weights for the trained neural network model.

[0178] In a further embodiment, the potential function is an elastic net.

[0179] One embodiment includes a computing system for training-concurrent quantization. The computing system includes a network interface; a memory storing instructions; and one or more processors configured to communicate with the network interface and the memory, the one or more processors further configured to execute the instructions to perform operations. The operations include obtaining a pretrained neural network model, wherein the pretrained neural network model comprises initial values fora potential function and a set of weights. The operations include training a neural network model based upon the pretrained neural network model and a set of training samples, wherein training the neural network model comprises, for each of a plurality of time iterations: calculating loss for a current neural network model having a current set of weights at a current time iteration using a loss function; updating a gradient of the potential function at the current time iteration according to a gradient of the loss; updating an inverse mapping of the gradient of the potential function; creating an updated neural network model including an updated set of weights for a next time iteration according to the gradient of the potential function and the inverse mapping of the gradient of the potential function, wherein creating the updated neural network model bimodally concentrates a distribution of the updated set of weights; and storing the updated set of weights. The operations include quantizing the trained neural network model after the plurality of time iterations, wherein quantizing the trained neural network model is performed according to a bimodality of the distribution of a trained set of weights.

[0180] In a further embodiment, the trained neural network model is a multi-layered neural network model.

[0181] In another embodiment, the multi-layered neural network model is selected from the group consisting of a convolutional neural network model, a recursive neural network model, a multi-layer perceptron, and a generative adversarial network.

[0182] In yet another embodiment, the multi-layered neural network model is a large language model.

[0183] In still another embodiment, the updated set of weights is further obtained based on at least one additional vector comprising a current set of auxiliary variables for the current neural network model.

[0184] In another embodiment, each sample of the set of training samples corresponds to a different auxiliary variable of the current set of auxiliary variables.

[0185] In still yet another embodiment, obtaining the updated set of weights for the next time iteration comprises obtaining an updated set of auxiliary variables based on the current set of weights for the current time iteration.

[0186] In another embodiment, each auxiliary variable of the current set of auxiliary variables is obtained based on a regularization parameter and the loss function evaluated for the current set of weights at the current time iteration.

[0187] In a further embodiment, a regularizer mirror descent (RMD) process is used to obtain at least one selected from the group consisting of the updated set of weights; and the current set of auxiliary variables.

[0188] In another embodiment, iterative updates are used to obtain at least one selected from the group consisting of the updated set of weights; and the updated set of auxiliary variables.

[0189] In still another embodiment, each of the current set of weights and the current set of auxiliary variables is initialized to at least one selected from the group consisting of: a zero vector and a non-zero vector comprising random numbers within a predetermined threshold of zero.

[0190] In yet another embodiment, the loss function comprises a constraintenforcing loss computed from the current set of auxiliary variables.

[0191] In a further embodiment, the constraint-enforcing loss is applied to prevent overfitting the neural network model to the set of training samples.

[0192] In another embodiment, a stochastic mirror descent (SMD) process is used to obtain the updated set of weights.

[0193] In still another embodiment, the loss function includes a Kurtosis term.

[0194] In yet another embodiment, the Kurtosis term is a function of the current set of weights.

[0195] In another embodiment, supervised learning is used to train the neural network model.

[0196] In a further embodiment, the loss function is configured to only equal zero when a predicted output value of the neural network model exactly matches a labeled output value from the set of training samples.

[0197] In still another embodiment, the loss function is selected from the group consisting of a quadratic loss function and a cross-entropy loss function.

[0198] In another embodiment, at the current time iteration, obtaining the updated set of weights is based, at least in part, on a learning rate at the current time iteration.

[0199] In yet another embodiment, the learning rate is time-invariant.

[0200] In a further embodiment, updating the gradient of the potential function at the current time iteration is performed using at least one system selected from the group consisting of Adam, AdamW, AdaGrad, RMSProp, Momentum, and WeightDecay.

[0201] In another embodiment, the gradient of the potential function at the current time iteration is obtained recursively based on the gradient of the potential function at a previous time iteration.

[0202] In still another embodiment, training the neural network model at the current time iteration further comprises storing the gradient of the potential function at the current time iteration.

[0203] In yet another embodiment, after the plurality of time iterations, the potential function is minimized via implicit regularization.

[0204] In another embodiment, the potential function is a strictly convex function.

[0205] In a further embodiment, the potential function is a norm function.

[0206] In still another embodiment, the potential function is selected from the group consisting of: an Lqnorm function where q » 1; and an £1+snorm function where 1 » E > 0.

[0207] In another embodiment, training the neural network model comprises at least one selected from the group consisting of distributed training, batch training, synchronous training, ensemble learning, meta learning, transfer learning, and continual learning.

[0208] In yet another embodiment, the inverse mapping of the gradient of the potential function is a many-to-one function.

[0209] In a further embodiment, a data size of each individual weight of the current set of weights is quantized to a single bit.

[0210] In another embodiment, the potential function is selected to enable the bimodality of the distribution of the trained set of weights.

[0211] In still another embodiment, the potential function is a rigid cage.BRIEF DESCRIPTION OF THE DRAWINGS

[0212] The description and claims will be more fully understood with reference to the following figures and data graphs, which are presented as exemplary embodiments of the invention and should not be construed as a complete recitation of the scope of the invention.

[0213] FIG. 1 conceptually illustrates an example of a network model configured in accordance with many embodiments of the invention to respond to user queries.

[0214] FIG. 2 conceptually illustrates an example of a process for training a model in accordance with an embodiment of the invention.

[0215] FIGS. 3A - 3C conceptually illustrate processes directed to the quantization of network models.

[0216] FIGS. 4A - 4B conceptually illustrate processes directed to the pruning of network models.

[0217] FIG. 5 illustrates a training-concurrent compression process implemented utilizing mirror descent in accordance an embodiment of the invention.

[0218] FIGS. 6A - 6B conceptually illustrate histograms of the absolute value of the final weights in a network model for different potentials when trained in accordance with various embodiments of the invention.

[0219] FIGS. 7A - 7B conceptually illustrate the impact that processes, implemented in accordance with some embodiments of the invention, may have on weight distribution for quantized systems.

[0220] FIG. 8 illustrates an example of a training system capable of training models using training-concurrent compression processes implemented in accordance with some embodiments of the invention.

[0221] FIG. 9 illustrates an example of a training element capable of executing instructions to perform training-concurrent compression processes that train and / or utilize models in accordance with miscellaneous embodiments of the invention.

[0222] FIG. 10 illustrates an example of a training application for training models in accordance with certain embodiments of the invention.DETAILED DESCRIPTION

[0223] Turning now to the drawings, systems and methods implemented in accordance with numerous embodiments of the invention may be used to simplify deep learning models including but not limited to neural networks and / or machine-learning (ML) models without sacrificing performance. Processes in accordance with a number of embodiments, are used to perform one or more compression processes including but not limited to pruning and quantization concurrently with model training. Models trained using training-concurrent compression processes can incorporate fewer parameters than models trained using conventional processes and / or achieve higher levels of performance compared to other models having similar numbers of parameters. In a number of embodiments, trained models may be deployed to server systems that are capable of generating inferences using the trained models. Training-concurrent compression processes and systems that utilize models trained using such processes (e.g., to generate inferences in response to user requests) in accordance with various embodiments of the invention are discussed further below.I. Background

[0224] Systems and methods in accordance with many embodiments of the invention may be used to facilitate training of neural networks implemented as structures including but not limited to generative artificial intelligence (“generative Al” or “gen Al”) models. Neural network models incorporated within generative AI systems implemented in accordance with certain embodiments of the invention may include but are not limited to large language models (LLMs) that are trained on large amounts of text data in order to perform natural language processing tasks including but not limited to deciphering, processing, and / or outputting textual phrases. As can readily be appreciated, however, the processes described herein are not limited to use with LLMs and references to LLMs throughout the disclosure should be understood as references to LLMs and / or other state of the art deep learning models. In particular, processes in accordance with several embodiments may be performed by any of a variety of classes of neural network models including (but not limited to) Convolutional Neural Networks (CNNs), Recurrent Neural Networks (RNNs), Multi-Layer Perceptrons (MLPs), and Generative AdversarialNetworks (GANs). The specific machine learning model that is utilized in systems implemented in accordance with various embodiments of the invention is typically dependent upon the requirements of a particular application.

[0225] A process for querying a machine learning model trained using a trainingconcurrent compression process in accordance with an embodiment of the invention is conceptually illustrated in FIG. 1. In the illustrated process, the machine learning model 120 is maintained by systems that receive queries 130 from user devices 140, and returns responses 145. In accordance with many embodiments of the invention queries may be received from users via user interfaces and / or be automated. Processes implemented in accordance with various embodiments of the invention may be used for machine learning (e.g., generative) models including but not limited to LLMs, diffusion models, and / or Hybrid models. Additionally or alternatively, in several embodiments of the invention, machine learning models 120 may be stored through various modes, such as remote storage and / or direct storage. For example, in some embodiments, machine learning models may be stored directly on user devices 140 used for generating (e.g., textual) queries 130. Additionally or alternatively, in certain embodiments, machine learning models 120 may be stored on remote servers accessed through network interfaces. In such cases, the user devices 140 may be configured to send queries 130 to machine learning systems that maintain machine learning models 120 over wired and / or wireless network connections such as Wi-Fi connections, cellular data connections, and / or the Internet.

[0226] Upon receiving user queries 130, machine learning system use the queries 130 to search a vector storage 150 for the most similar entries 160. In several embodiments, for example, when the machine learning model 120 is an LLM, the queries 130 may be converted from their original (e.g., textual) forms to vector 170 forms for the search. In various embodiments of the invention, the vector storage 150 may be configured to return (and / or the query vectors 170 may be configured to request) that a certain number of most similar entries 160 be returned to the machine learning system. The most similar entries 160 may have a quantity that is predetermined. In several embodiments, the similarity of the most similar entries 160 may be determined by, but is not limited to, comparing the vectors (e.g., 130, 170) to determine cosine similarity. Oncereturned, the most similar entries 160 may be converted, by the machine learning system, from vector forms to (e.g., textual) forms for user review. Once done, these (e.g., textual) responses 145 may be returned to the user devices 140.

[0227] In accordance with many embodiments of the invention, machine learning models 120 may be trained based on training data 190 (and / or samples). The training data 110 implemented may include (but is not limited to) video, audio, text, images, etc. In various embodiments, training data 190 may include labels (e.g., in the form of scalars) classifying the substantive input training data. Training data 190 in accordance with some embodiments of the invention can be received continuously by the machine learning system, where the network model(s) 120 may be continuously updated as new training data 190 is received. Based on the training data 190, the machine learning model 120 may convert the training data 190 into embeddings 180 and / or feature vectors that (e.g., numerically) represent the underlying information (classifications). In several embodiments of the invention, the embeddings 180 may be kept in vector storage 150. In some cases, the vector storage 150 may be kept but is not limited to user devices 140 (e.g., where the machine learning model 120 is stored) and / or remote servers (e.g., that may be stored along with the machine learning model 120). Training in accordance with numerous embodiments of the invention can result in models that are capable of generalizations to new unseen samples, as is expanded upon below.

[0228] An example of a training-concurrent compression process for training a machine learning model in accordance with an embodiment of the invention is illustrated in FIG. 2. In a variety of embodiments, processes can be performed in parallel, across multiple processors and / or computers. Process 200 initializes (205) a machine learning model. Machine learning models in accordance with numerous embodiments of the invention can include (but are not limited to) neural networks, such as CNNs / ANNs, linear models, etc. In several embodiments, initializing a model can include pre-training the model on a first set of data, where the training process can use a different second set of data for transfer learning. Initializing the model in accordance with various embodiments of the invention can include pre-training the model to have loss at or under a given threshold (e.g., 0). As can readily be appreciated, the use of a pre-training process and / orthe specific pre-training process utilized (if at all) is largely dependent upon the requirements of a given application.

[0229] Process 200 accesses (210) a set of training samples. Training samples in accordance with several embodiments of the invention can include labeled data. In some embodiments, training samples can include various data types (and / or accompanying labels), such as, but not limited to, images, video, text, numeric data, etc. In many embodiments, the set of training samples can be applied to training the model concurrently with the process 200.

[0230] Process 200 computes (215) a loss component for a loss function that calculates losses for the set of training samples. As discussed above, in some embodiments, the model may be a neural network undergoing supervised learning or unsupervised learning. In the former case, the loss component may be configured to measure the differences between a labeled value for one or more training samples and the predicted values for the samples. In many embodiments, loss components can include a constraint-enforcing loss. Constraint-enforcing losses in accordance with several embodiments of the invention can be used to prevent overfitting of the model to the training samples. In many embodiments, constraint-enforcing losses can be computed from a set of auxiliary variables, that may refer to (e.g., neural network) parameters. Additionally or alternatively, in various cases, each sample may have at least one associated auxiliary variable. Auxiliary variables in accordance with many embodiments of the invention can be updated based on a gradient of the constraintenforcing loss computed for one or more training samples. In several embodiments, auxiliary variables can be initialized to 0 and / or to random values (near-zero). When auxiliary variables are random values initialized to near-zero, they may be initialized using a random number generator (e.g., within a pre-determined threshold of zero).

[0231] Process 200 applies (220) a potential function to weights of the model to compute a regularizing component of the loss function. Potential functions (“potentials”) in accordance with a variety of embodiments of the invention can include various q-norm potentials, where q is a number (e.g., 1, 2, 3, 10, etc.), and / or a negative entropy potential. In certain embodiments, processes can select potential functions to achieve certain objectives (e.g.,norms to promote sparsity, ℓ10norms to promote generalization, etc.).Regularizing components in accordance with certain embodiments of the invention can be selected to optimize closeness to the initialized model, where closeness can be computed as a Bregman divergence.

[0232] Process 200 updates (225) weights of the model based on the determined losses for the set of training samples. Although this process is described as a single iteration, one skilled in the art will recognize that similar systems and methods will generally be used as part of an ongoing optimization process. In a number of embodiments, optimizations can be performed with weighted values to emphasize the loss or regularizing components of the loss function. Processes in accordance with many embodiments of the invention can weight the loss component for an overparameterized model such that the loss is forced to 0, where the process further optimizes the regularizing component.

[0233] While specific processes for training a model are described above, any of a variety of processes can be utilized to train models as appropriate to the requirements of specific applications. In certain embodiments, steps may be executed or performed in any order or sequence not limited to the order and sequence shown and described. In a number of embodiments, some of the above steps may be executed or performed substantially simultaneously where appropriate or in parallel to reduce latency and processing times. In some embodiments, one or more of the above steps may be omitted. In addition, models trained in accordance with the methods described above with reference to FIG. 2 can be utilized in any of the systems described herein.

[0234] In implementing the above systems, potential functions and / or constraintenforcing losses may be used to facilitate the training of the deep learning models. The alignment of a model and a data point i can be measured by a (non-negative) “loss function” L (w) (or ■ / ’(•,•)) for any weight vector w. For a training set consisting of n data points, the total loss can be calculated as ∑ᵢ₌₁ⁿ Lᵢ(w), which can then be minimized (e.g., using processes in accordance with many embodiments of the invention. In many cases, minimization is typically done using an algorithm called stochastic gradient descent (SGD) (or its variants, such as distributed, mini-batch, adaptive, and momentum). Denoting the model parameters at the t-th time step by wt, and the instantaneous loss function corresponding to the i-th sample by L (-), the update rule of SGD is defined as wt=wt-i ~ ij Li wt-i t > 1, where TJ is a hyper-parameter known as the “step size” or the “learning rate,” w0is the initialization, and VL;(-) is the gradient of the loss (usually computed using an approach known as “backpropagation”). This procedure can be repeated many times until some stopping criterion is met. As can readily be appreciated SGD is simply one approach to minimizing a total loss calculation and the specific total loss function and minimization algorithm that is utilized is largely dependent upon the requirements of specific applications.

[0235] Deep learning neural network models directed to purposes including but not limited to natural language processing tasks (e.g., LLMs) can have as many as several trillion parameters. The memory, computing, storage, and communication costs required for models of this size can make deep learning models both exceedingly expensive (both financially and temporally) and especially difficult to personalize and / or integrate into low-latency / real-time applications. Systems may address expenses associated with these amounts of parameters through simplification processes including but not limited to (data) compression. The effects of the above simplifications can include saving memory and speeding up inference speed. As such, the term compression is used herein to describe any processes directed to the objective of reducing the memory (and / or processing power) associated with running machine learning models. As such, i accordance with many embodiments of the invention, compression processes may involve but are not limited to network quantization (i.e., representing weights and activations of an existing network with lower bit precision) and network pruning (i.e., removing parameters from an existing network). Many embodiments of the invention are able to achieve significant compression with minimal impact on model performance by using one or more of the various training-concurrent compression processes described below.

[0236] An example of the quantization of a neural network is illustrated in FIGS. 3A - 3C. Quantization may refer to the process of reducing the precision of (and memory requirements associated with) network parameters including but not limited to weights, biases, and / or activations such that the quantized network parameters occupy less memory. Most commonly, quantization involves reducing 32-bit or 16-bit binary representations of floating point values (e.g., the example disclosed in the various weights of FIG. 3A) to smaller representations in the form of 8-bit binary representations of integervalues. Nevertheless, conventional quantization processes generally involve quantizing fully-trained networks, which carries the possibility of cascading effects where quantization of especially salient parameters (i.e., parameters for which the model output is especially sensitive) significantly impact the efficiency and / or efficacy of the neural network.

[0237] Systems and methods in accordance with many embodiments of the invention perform training-concurrent quantization processes that quantize parameters to a single bit representation, which can be referred to as a one-bit quantization process in this disclosure. As suggested above, one-bit quantization in multi-layered neural networks, e.g., convolutional neural networks (CNNs), recursive neural networks (RNNs), and LLMs, can provide significant memory storage and computation benefits. In such networks, at each layer, the output of the activation functions may be “weighted” by the model parameters and summed to form the inputs to the activation functions in the next layer. When the parameters are floating point numbers, this “weighting” is performed using multiplication operations. When the weights are binary (single bit) multiplication is unnecessary and the necessary operations may be implemented using addition and subtraction operations. In accordance with various embodiments of the invention, one-bit quantization of a neural network model may be achieved through quantizing the models as the model is trained. Specifically, one-bit quantization may be performed by starting with a large network model and simultaneously learning from the training data while guiding the weights towards a distribution that is much more amenable to quantization. As shown in FIG. 3B, the weights are updated in such a way that the values of the parameters in each layer concentrate to one of two values: +a and -a. This makes quantization to one-bit straightforward binary reduction — the weights concentrated around +a are quantized to “1,” and the weights concentrated around -a to “0,” as illustrated in FIG. 3C.

[0238] An example of the pruning of network models is illustrated in FIGS. 4A -4B. Pruning may refer to the process of removing redundant network parameters including but not limited to weights, biases, and / or activations from neural networks as a form of compression. Most commonly, pruning may be used to combat overfitting and improve generalization capabilities. As is the case for quantization, the salience ofparticular network parameters may be evaluated before determining whether they should be pruned. Many implementations of pruning employ some second-order information about the error landscape for this purpose, an approach which can be very costly to implement and often does not scale to large models (in particular, LLMs) due to similar cascading effects that can occur during conventional quantization processes. Therefore, systems in accordance with various embodiments of the invention may be used to facilitate network pruning in tandem with learning to produce systems that are more amenable to large-scale operation. By doing so, pruning may be used to reduce the size of over-parametrized networks without sacrificing too much performance. Systems and methods in accordance with various embodiments may, additionally or alternatively, monitor the performance (or training efficacy) of networks undergoing pruning. In such cases, systems may be configured to cease pruning beyond a threshold during the training process.

[0239] As described above, systems and methods in accordance with many embodiments may prune networks as they train, starting with a large network model and simultaneously learning from the training data while removing weights that are redundant and / or unnecessary. By doing so, resulting neural networks will have been pruned to find a set of weights that performs well on the training data while being sparse. This is illustrated in FIGS. 3A and 3B of the filed application, where FIG. 3A shows a fully connected neural network with all weight connections intact between layers, and FIG. 3B shows the same network architecture after pruning where certain connections (weights) have been removed, resulting in a sparser network structure with fewer parameters while maintaining the essential connectivity needed for the model to function.

[0240] Applications and methods in accordance with various embodiments of the invention are not limited to neural networks. Accordingly, it should be appreciated that the data compression processed described herein can also be implemented with various model configurations in accordance with the invention. Various systems and methods for implementing training-concurrent compression processes in accordance with some embodiments of the invention are discussed further below.

[0241] Systems in accordance with many embodiments of the invention may use learning to facilitate decision-making processes. Supervised learning uses (labeled)training data to build mapping functions that associate new data with expected output values for purposes including but not limited to pattern recognition and / or prediction. As such, in accordance with some embodiments of the invention (e.g., supervised) learning may use training data sets {xi,yi}i=1n, where n represents the size of the training set; xi∈ ℛdinrepresents the feature vector (i.e., input); dinrepresents the dimension of the feature(s) (e.g., the number of pixels in image input; the context length in text input);yi∈ ℛdoutrepresents the corresponding label (i.e., output); and doutrepresents the dimension of the label (e.g., the context length in label text). In some embodiments of the invention, labels may be scalars (i.e., dout= 1).

[0242] Network models implemented in accordance with many embodiments of the invention may map variables including but not limited to the feature vectors xi; weight vectors w ∈ ℛp(where p represents the size of the weight vector, i.e., the number of parameters in the model); and / or various input-associated variables to the labels ytvia a function / (-,•). In accordance with many embodiments of the invention, the number of parameters p well exceeds the size of the training data n, i.e., p > n (overparametrization). In such cases, network models may perfectly interpolate (learn) the training data, determining that there exists at least one optimal weight vector w, such that:f(xi, w) = yi, i = 1, 2, ..., n.(1) In fact, the set of weight vectors w that interpolate the data may form a p - n dimensional manifold which, when p » n, becomes very high-dimensional. Systems and methods in accordance with certain embodiments of the invention may exploit the high-dimensional interpolating manifold(s) to find solutions that is simultaneously interpolating and primed for compression. Specifically, by training in mirrored domains and / or utilizing (constraintenforcing) losses, processes in accordance with some embodiments of the invention can train models to achieve various objectives, such as (but not limited to) sparseness and / or generalization. Processes in accordance with various embodiments of the invention can therefore be used for training networks in various settings including, but not limited to: distributed learning, (mini)batch learning, (a)synchronous learning, ensemble learning, meta learning, transfer learning, and continual learning (and / or learning with features including but not limited to adaptive learning rates, momentum, and / or early stopping).II. Mirror Descent Algorithms

[0243] Systems and methods in accordance with various embodiments of the invention can improve optimization processes significantly through implementing mirror descent algorithms.A. Background

[0244] Systems and methods in accordance with various embodiments can train neural networks by solving various optimization problems. Systems can, in some cases, augment a loss function with a term that promotes closeness to the initial weights w0and train the neural networks by solving the following optimization problem:n min L((w) + D^,(w, w0), (2)t=iwhere is the Bregman divergence corresponding to a differentiable strictly-convex function i -. K, referred to as the “potential function.” For example, when (w) = - II w II2, the Bregman divergence is just the usual Euclidean distance, i.e., D^(w,w0) = - II W - WQ II2. Other examples of potential functions in accordance with several embodiments of the invention are discussed in greater detail below. In certain embodiments, models can be “warm -started” with an initialization weight w0, e.g., for transfer learning. In some such cases, a parameter A can determine how much weight one wants to give to the loss versus a “regularizer” (where the bigger A is, the more effort is spent on minimizing the loss). In scenarios where a particularly good initialization w0is not known, and / or where it is desirable to regularize the weights in an absolute sense, systems in accordance with some embodiments can choose w0to be the minim izer of(e.g., 0 for = | II w II2and other potential functions). In this case, the optimization problem (2) can be reduced to the following special case:nmin A? LI(W) + ip(w). (3) i=l

[0245] Typical deep neural networks can often have a lot of capacity (i.e., a large number of parameters), which allows them to fit the training data to zero error (i.e., L((w) ® 0). However, for various reasons, e.g., when a training data set includescorrupted samples, it may be desirable to avoid fitting the training data all the way to zero error / loss. That is part of the reason the above formulations can be beneficial in many applications.

[0246] In many embodiments, auxiliary variables can be used to avoid fitting data to zero error / loss. Defining an auxiliary variable z E H with elements z(i) for i = 1,...,n, optimization problem (2) can be transformed into the following form:Vz2(0 ™n^ — + ^(*■“'0)(4)s. t. z(i) =y / 2Li(w'), i = 1,..., n.Gwi r^oix L U I,Z J IQ J / z11 = ip(w) +- II z lr. Because the objective is a Bregman divergence, andthere are n equality constraints, processes in accordance with a variety of embodiments of the invention can derive (mirror descent) algorithms for solving them, as described in greater detail below. In many embodiments, in order to enforce the constraints z(i) = 2Lj(w), a “constraint-enforcing” loss can be defined as / '(z(i) -yj2Li(w')>), where ^(-) is a differentiable and convex function with a unique root at 0 (an example is ^(-) = ^-).

[0247] In several embodiments of the invention, at time t, when the i-th training sample is chosen for updating the model, the following update is performed:„.(,n.(., (0 - v'WZi)). V^(wt) =V2Li(wt-l)>...... - 72Li(wt-1)) Zt(l) = Zt-iCO - - -, Azt(fi = * i, (5) where V^(-) 'sthe gradient of the potential function, and ^'(-) is the derivative of the constraint-enforcing loss function. The variables can be initialized with w0and z0= 0 (or something close to 0). In accordance with many embodiments, in cases where, due to practical considerations, weights and / or the auxiliary variables cannot be initialized at zero, they can be initialized randomly at some small values (e.g., within a certain threshold of zero) without impacting performance. Note that because of strict convexity of thepotential function > C), its gradient Vi (’) 'sinvertible, and the above update rule is well-defined. This iterative process can solve the optimization problem (2) (and the optimization problem (3) when w0= 0). y parameterB. Stochastic Mirror Descent and Reqularizer Mirror Descent

[0248] Mirror descent algorithms including but not limited to stochastic mirror descent (SMD) and regularizer mirror descent (RMD) algorithms may be used to simultaneously learn and prime neural networks configured in accordance with various embodiments of the invention for compression during the training process. In doing so, systems may (e.g., continuously update values for weights and / or auxiliary variables. These algorithms may operate based on functions including but not limited to the (strictly convex) potential function,? (■), of the weight vector w GP, where p is the number of parameters in the network; and / or loss functions / ’(•,•)■ Ascanbe readily appreciated, the specific functions that are utilized in an SMD and / or RMD algorithm are largely dependent upon the nature of the network being optimized and / or the requirements of specific applications.

[0249] A process for training-concurrent compression, implemented utilizing mirror descent in accordance with some embodiments of the invention is illustrated in FIG. 5. Process 500 pretrains (505) a network model on a set of training samples. Models in accordance with numerous embodiments of the invention can include (but are not limited to) neural networks, linear models, etc. In several cases, pretraining (505) may include steps for determining initial values for weights (i.e., w0) and / or auxiliary variables (i.e., z(0)) Pretraining (505) the network model can involve, but is not limited to using datasets (including but not limited to the training dataset incorporating the set of training samples) for transfer and / or supervised learning. Meanwhile, training samples in accordance with several embodiments of the invention can include labeled data. In miscellaneous embodiments, training samples can include various types of data and / or labels, such as (but not limited to) images, video, text, numeric data, etc.

[0250] Process 500 calculates (510) loss for the network model at the current time iteration using a loss function. Examples of loss functions implemented in accordance with several embodiments of the invention may include but are not limited to quadratic loss and cross-entropy loss. In accordance with various embodiments of the invention,loss functions • (•,•) may be configured such that, when a predicted output value exactly matches a labeled value, the loss is determined to be zero (i.e.,= 0, iff a = b). As can readily be appreciated, the specific loss function that is utilized is largely dependent upon the requirements of specific applications.

[0251] Process 500 updates (515) the gradient of the potential function (of the network model) at the current time iteration according to a gradient of the loss. Potential functions in accordance with numerous embodiments of the invention may include but are not limited to (e.g., strictly) convex functions. Additionally or alternatively, potential functions can include various q-norm potentials, where q is a number (e.g., 1, 2, 3, 10, etc.), and where the various values for q correspond to different process 500 objectives including but not limited to types of data compression.

[0252] Process 500 updates (520) the inverse gradient of the potential according to the gradient of the loss. The gradient of the potential vector can be viewed as a mapping from the weight space of w to the gradient space (the mirror domain). As such, in accordance with various embodiments of the invention, for each iteration (t), SMD can operate under the assumption that the gradient of the potential function would be updated along the negative direction of the loss function to train the model. In various embodiments of the invention, updating the inverse gradient may be done based on parameters including but not limited to (e.g., time-varying) step size (e.g., based upon learning rate). Process 500 updates (525) the weights for the next time iteration according to the gradient of the potential and the inverse gradient of the potential. In accordance with many embodiments of the invention, the updated weights may be used for data compression purposes. As suggested above, when the form of data compression corresponds to a pruning process, each update (525) to the weights may involve a specific pruning step. Additionally or alternatively, when the form of data compression corresponds to a quantizing process, each update (525) to the weights may be directed to producing a final bimodal weight vector that can subsequently be used for (e.g., one-bit) quantization.

[0253] While specific training-concurrent processes for updating model weights using mirror descent are described above, any of a variety of processes can be utilized to train models as appropriate to the requirements of specific applications in accordancewith various embodiments of the invention. In several embodiments, steps may be executed or performed in any order or sequence not limited to the order and sequence shown and described. In numerous embodiments, some of the above steps may be executed or performed substantially simultaneously where appropriate or in parallel to reduce latency and processing times. In multiple embodiments, one or more of the above steps may be omitted. Furthermore, the training concurrent processes are not limited to the specific loss and / or potential functions described above and can utilize any loss and / or potential function appropriate to the requirements of specific applications.a. SMD Process for Compression

[0254] In accordance with many embodiments of the invention, SMD may facilitate model compression alongside model learning using systems implemented to update the gradient of the potential vector with respect to the (current) weight vector. SMD may therefore update the gradient of the potential, Vi (w(ty), along the negative direction of the loss function with respect to the model parameters. In several embodiments of the invention, this update can operate according to the following equation:Vi / >(w(t + 1)) = Vi / >(w(t)) - 77(t)V^( -(x(t), w(t)),y(t)), w(0) (6) where t is the iteration, w(t) is the weight at iteration t, and (x(t),y(t)) is the training point used at iteration t. In this equation, V represents the gradient with respect to the weight vector, so that V^(-)ar|d V-Z’(-r) are the gradients of the potential and loss functions, respectively (e.g., with respect to the weight vector). 77(f) > 0 is the learning rate at iteration t. It may be time-varying or time-invariant, i.e., 77(f) =?. Finally, w(0) is the value, to which that the weight vector is initialized. In accordance with many embodiments of the invention, the gradient update term(s) [e.g., 77(f) V- / 1w(t)), y (t) ) ) may be replaced by variations, including but notlimited to (ones based on) Adam, AdamW, AdaGrad, RMSProp, Momentum, WeightDecay, etc. As can readily be appreciated, the specific manner in which the gradient update terms are calculated is largely dependent upon the requirements of specific applications.

[0255] Among all solutions that interpolate the training data, SMD may be used to converge on one or more solutions that minimize the potential ^(-). As mentioned earlier, in the over-parametrized scenarios, there are infinitely many weight vectors that satisfy the interpolating conditions. In accordance with many embodiments of the invention, when initialized by w(0) = w0, after convergence, SMD may find an interpolating solution that satisfies:min ip(w) - \7xp(wo)T(w - w0). w such that (Xi,w)=yi,i=l,2,...,n (7)

[0256] As mentioned above, for overparameterized linear models, SMD can converge to the closest global minimum to the initialization point(s). For initialization points around “zero” (i.e., the minimizer of the potential), the closest global minimum to the initialization point may mean convergence to the minimum-potential interpolating solution. Specifically, when w0is initialized as the minimizer of ip, so that Vi (w0~) = 0, after convergence, SMD may find a solution that satisfies:min V'(w) (8) wsuch that Cxl,w')=y(,t = l,2,...,ri a process known as implicit regularization.

[0257] Since the learning algorithm is a function of the updated weights, an inversion (depending on an inverse mapping of the gradient of the potential: V-1^(-)) may be calculated for the actual process of updating the weights based on equation (6). This may be implemented in at least two variations: a one-step process with a lower memory requirement, but potentially lower numerical stability / consistency; and a two-step process with a high memory requirement, but higher numerical stability.

[0258] The one-step process for updating weights may operate based on computing both the gradient of the potential and the inverse gradient of the potential at each iteration. In various embodiments, for a given time step, updates to the weights may similarly be based on, but are not limited to:• estimates for the (current) gradient of the potential (i.e.,Vi / >(w(t)));• training inputs (i.e., x(t));• corresponding training output values (i.e., y(t));• (current) weight values (i.e., w(t));• predicted output values based on the training inputs / weights (i.e., / (x(t),iv(t)));• loss gradients, based on the loss computed between the predicted output values and the training output values (i. e., ( ).y( )); and• learning rates (i.e., 77(f))For example, one estimate for future weights may effectively follow the following:w(t + 1) = [vV>(w(t)) - r](t)W ( / (x(t),w(t)),y(t))], w(0). (9)The one-step update (9) has the advantage that one needs only store the weight estimate w(t) at each time iteration. In some cases, these operations may be numerically ill-conditioned which, as mentioned, can affect the numerical stability of the method.

[0259] The two-step method may alleviate the aforementioned numerical instability by computing the gradient of the potential at the point of initialization V >(w(0)) and continuously / recursively deriving and storing estimates for the gradient of the potential (at later time iterations) based on that initial estimate. These estimates for the future gradient of the potential may, in turn, be used to update the weights. In some embodiments, for a given time step, estimates for the future gradient of the potential may similarly be based on, but is not limited to: estimates for the (current) gradient of the potential; training inputs; corresponding training output values; (current) weight values; predicted output values based on the training inputs / weights; loss gradients, based on the loss computed between the predicted output values and the training output values; and learning rates. That said, in some cases, when the estimates for the gradient of the potential is defined as ((t) = V^(w(t)) the process at each iteration may be approximated as:U(t + 1) = ( (t) - 77 (t) ( / (x(t), w(t)), y(t)}, ((0) = Vtp(w(0) )( w(t -I- 1) = + 1)] The two-step method can substantially alleviate numerical instability since it only requires computing inverseat each iteration. The price for this is that the memory requirement doubles: systems will need to store both ((c) and w(t).b. RMD Process for Compression

[0260] Additionally or alternatively, systems configured in accordance with some embodiments of the invention may facilitate model learning using RMD to update the gradient of the potential vector with respect to other vectors (i.e., auxiliary variables). In such embodiments, in addition (or alternative) to updating the weights w(t) in the mirror domain, systems may update a set of n auxiliary variables zi(t). In some cases, the weights may be interpreted as just another variation on the auxiliary variables, being updated in a comparable manner. For example, each of these auxiliary variables may themselves correspond to particular element(s) of the training set {%j,y ”=1.z.(t +l)=if i = i(t)I z;(t) otherwiseHere i(t) may refer to the index of the training element that is being used for the gradient updates at time t; and A > 0 may refer to a regularization parameter.

[0261] As above, for each time step / iteration (t), RMD can operate under the assumption that the gradient of the potential (with respect to the weight vector) would be updated along the negative direction of the loss function to train the model. The update rule, in such cases, may assume that updating the gradient of the potential is based, not only on the loss function and learning rate, but on the auxiliary variables. This may be done as follows: / i - \2Vip(w(t + 1)) = V?p(w(t) ) - j?(t)V (f w(t)}, yiW^ j,w(0) = w0, z(0) = 0 (12)In accordance with some embodiments, the gradient update term(s) ^e.g.,rj(t)v(ziW)-)2 / i - \\,^(0 \ (f w(t)),yi(t)) H may be replaced byvariations, including but not limited to (ones based on) Adam, AdamW, AdaGrad, RMSProp, Momentum, WeightDecay, etc. As can readily be appreciated, the specificmanner in which the gradient update terms are calculated is largely dependent upon the requirements of specific applications.

[0262] Among all solutions that interpolate the training data, RMD may similarly be used to converge on one or more solutions that minimize the potential i>(-). When RMD is initialized with w(0) = w0, the minimizer of?;(•), and z(0) = 0, the weight vector may converge to the solution of:n min ^( / (Xj,w),yj + A^(w) (13) wLi=i where A > 0 is the regularization parameter mentioned earlier in the RMD recursions (explicit regularization). In doing so, determines the trade-off between how much we want to reduce the loss?=i^CfCxt>w>yi) versus how much we want to reduce the regularizer i^(w).

[0263] As mentioned above, an inversion (depending on an inverse mapping of the gradient of the potential V-1^(-)) may be calculated for the actual process of updating the weight vector w(t) itself (in addition to the vector(s) of auxiliary variables). Furthermore, as above, these updates may be implemented in at least two variations: a one-step process and a two-step process. The one-step process may follow the equation: / i - \ 21W(t + 1) =1Vl / >(w(t)) - J](t)V I Zj(t) -w(0) = w0, z(0) = 0 (14) Meanwhile, as above, the two-step method may alleviate the aforementioned numerical instability by computing the gradient of the potential at the point of initialization ( V^(w(0))) and continuously deriving and storing estimates for the gradient of the potential (at later time iterations) based on that initial estimate (in addition or alternative to variables including but not limited to the loss function and / or learning rate). The two-step process may thereby follow the equation:( I - \2ziM ~ (xi(t)Mt)),yKt)) j,f(0) = Vi / >(wo),z(0) = 0 w(t + 1) = + 1)).

[0264] In accordance with multiple embodiments of the invention, RMD may be considered a broader instance of SMD. Specifically, as the regularization parameter of RMD, 2 ^ 0, the implicit regularization of SMD may be recovered. This can also be seen from the recursions for RMD (11-15) which reduce to their SMD counterparts by setting 2 = 0. When applied to network compression, the advantage of RMD over SMD is that systems implemented in accordance with certain embodiments of the invention may perform more aggressive simplifications by further reducing the value of the potential function! / >(•)■ Insome instances, this may significantly improve accuracy. That said, this improvement may come at the price of slightly sacrificing on the training loss.c. Special Cases for Different Potential Functions

[0265] In accordance with various embodiments of the invention, potential functions may vary in implementation. Specifically, distinct potential functions may be applied in order to optimize specific outcomes. Several variants on potential functions implemented in accordance with some embodiments of the invention are disclosed below.i. q-norm Potential

[0266] An important special case, for applications of mirror descent algorithms utilized in accordance with some embodiments of the invention, is when the potential function >(-) is chosen to be the -£qnorm, frequently chosen as the potential function in order to optimize regularization. In doing so, the fqnorm may take the form: >(w) = ~ II w llq = ~^k=i |w(fc)|q, for any positive integer q. Note that this choice of potential function is “separable,” in the sense that the update for the fc-th element of the weight vector requires only the k-th element of the weight and gradient vectors. For example, let the current gradient be denoted by g-. -In this case, the update rule can be written as:. n-e'tzt-ifi) — _L_ = I Iwt-iC / c) P ^iqnCwt-iCfc)) +- - - -gtjt1x. — d2Li(wt--S) sign^wt-^k^1sign(wt-1(k)') + - - — ^(fc)), V / c V2Li(wt-i)Zt(l) = Zt-lV - ^'(zt-AV ~ V2Li(wt-l))<zt(j) = Zt- j. Yj * i, (16) where wt(k) denotes the k-th element of wt(the weight vector at time t), and g(k) is the k-th element of the current gradient g. This allows for efficient (parallel) implementation (-.of the algorithm. In certain embodiments, ^(-) = which implies '(•) = (•) andsimplifies the updates:. g(zt_.(i') — j2Lj(wt_S} _L_ wt(fc) = | |wt-1( / c)|'7 lsign(wt-1(k)) +- - - -gWt1x V2£i(wt-i)■ n *l(zt-i(i) - V2Li(wt-i))w. sign^Wf.^k)^1sign(wt-1(k^ + - - g(k)), Vkzt(i) = Zf-M - j(zt_i(i) - g2Li(wt-1^),zt(j = Zt-r j. Yj * i, (17)

[0267] Implementing compression-directed functionalities can modify how network models in accordance with various embodiments of the invention are primed. Moreover, in a number of embodiments, processes can use different q for different effects as t -norm regularization can have a different effect on the weights of the corresponding q. This can be especially impactful when performing compression processes implemented in accordance with some embodiments of the invention.

[0268] Histograms of the absolute value of the final weights in the network for different potentials are illustrated in FIGS. 6A and 6B. In this example, the solutions obtained by different SMD algorithms are vastly different from one another, even though they all fit the same training data and even though they were initialized with the same set of weight vectors, which highlights the role of the proposed algorithm for training. Each of the histograms corresponds to an 11 × 106-dimensional weight vector that perfectly interpolates the data.1. Quantization-directed -Norm Applications (fm)

[0269] In applying ( / -norm potentials to quantization, (strictly) convex potentials such as:p V> O) - ||w||; = (18)i=l for q » 1, encourage the weights to concentrate around two bipolar values. Thus, running SMD (and RMD) for ip(w) = || w||qq, for q » 1, when initialized by w0= 0 (or some weight vector close to zero), typically results in a final weight vector that is very close to bimodal. FIGS. 7A and 7B illustrate the effect that these training processes can have on network models implemented in accordance with numerous embodiments of the invention. Specifically, FIG. 7A effectively illustrates a weight distribution, corresponding to (one layer) of an LLM trained using standard processes. This distribution is distinctly unimodal and is shown to have a mode centered at (or near) zero. In contrast, FIG. 7B illustrates a layer of an LLM trained based on methods in accordance with multiple embodiments of the invention. The bimodal distribution, with modes centered roughly at weight values of roughly ±0.3, is far more easily quantized to a single bit (for example, one where the weights centered near -0.3 are quantized to “0” and those centered near 0.3 are quantized to “1”).

[0270] This configuration is especially effective when systems are priming networks for quantization. Specifically,norm regularization promotes bounded and small ranges of weights. With this choice of potential, the weights tend to concentrate around small intervals. This is often desirable in many implementations of neural networks since it can provide a small dynamic range for quantization of weights, which can reduce the production cost and computational complexity. However, since is not differentiable, processes in accordance with some embodiments of the invention can use a large valuefor q, e.g., q = 10 and implement i^(w) =II w ||]§ to achieve the desirable regularization effect ofThe histogram of the ^10SMD in FIG. 6A illustrates this regularization effect in practice. Specifically, after extended training, the distribution on the corresponding figure has completely moved away from zero, i.e., all the weights in the £wsolution are non-zero. This emphasizes how the quantization-optimizing approach corresponding to ^10, which uses the entire overparameterization available in the network, generalizes especially effectively.

[0271] In several embodiments, when the potential is ip(w) = || w||^, systems configured in accordance with multiple embodiments of the invention may determine:[Vt / > O)]i = qWilWil^-2= qlwJ^signCw;) (19) where [V^(w)]fand wzare the ith components of the vectors VI / J(W) and w, respectively. In such cases, the inverse of the gradient of the potential may be given by:2—q 1[[∇⁻¹ψ(x)]ᵢ = (|xᵢ| / q)^(1 / (q-1)) · sign(xᵢ). (20)By using the above (^ norm-directed) equations (19) and (20) with the inverse mapping (of the gradient of the potential) required for the weight update processes of equation (9) for SMD (or equation (14) for RMD), systems in accordance with some embodiments may implement the one-step process for SMD with anorm. By doing the same with equation (10) for SMD (or equation (15) for RMD), systems may implement the two-step process. Nevertheless, in the latter case, as expressed above, the need for equation (19) (i.e., [V (w)],) may be limited only to the time iteration of initialization (e.g., while equation (20) is used at every iteration).2. Pruning-directed q-Norm Applications (ℓ₁)

[0272] In contrast, where q = 1 + e, 1 » e > 0 (i.e., 2 » q > 1), SMD (and / or RMD) is running for i (w) = ||w||i+f, and the corresponding model initialized by w0= 0 (or some weight vector close to zero), systems implementing processes in accordance with a number of embodiments may result in a final weight vector that is sparse. Specifically, norm regularization promotes sparsity in the weights, which is often desirable for reducing the storage and computational load, since deep neural networks often have millions or billions of weights. However, since thenorm is not differentiable or strictly convex, processes in accordance with many embodiments of the invention can useIIwlli+f for some small e > 0. The histogram of the / ’1SMD in FIG. 6B illustrates the effect of this form of training on the sparsity of the weight distribution. Specifically, the histogram of the / '1-SMD in FIG. 6B has more weights at and around zero, i.e., it is very sparse.

[0273] Moreover, when the potential is i (w =|| systems configured in accordance with some embodiments may determine:W;[Vi / v(w)], = (1 + e)-j — = (1 + e)|wdesign(wf), (21)where [V^(w)]iar|d are the ith components of the vectorsar|d w, respectively. In such cases, the inverse of the gradient of the potential may be given by:[∇⁻¹ψ(x)]ᵢ = (|xᵢ| / (1+ε))^(1 / ε) · sign(xᵢ). (22)By using the above (ℓqnorm-directed) equations (21) and (22) with the inverse mapping (of the gradient of the potential) required for the weight update processes of equation (9) for SMD (or equation (14) for RMD), systems in accordance with various embodiments may implement the one-step process for SMD with anorm. By doing the same with equation (10) for SMD (or equation (15) for RMD), systems may implement the two-step process. Nevertheless, in the latter case, as expressed above, the need for equation (21) (i.e., [∇ψ(w)]ᵢ)may be limited only to the time iteration of initialization (e.g., while equation (22) is used at every iteration).3. Joint Compression

[0274] Systems and methods in accordance with various embodiments of the invention may use multiple compression processes in tandem. For example, systems may be configured to jointly prune and quantize the parameters in a neural network. As suggested above, taking the potential as the1+enorm, with a small e > 0, encourages the parameters to be sparse, and hence prunable. Therefore one possible framework (to allow SMD and / or RMD) to jointly prune and quantize is to take:ψ(w) = max(|wᵢ|^(1+ε), β|wᵢ|^q) (23)t=l where 1 >> ε > 0, q >> 1, and β > 0 is a constant that weighs the relative norm potentials.

[0275] When the potential follows the above configuration, systems configured in accordance with many embodiments may determine:1l\q-l-e (1 + f)k,|e• sign(wj)1 l\q-l-e [V^(w)]f= [(1 + e)|w / |e,y? Q|wd(?^signOi) (24)1 l\q-l-e Pq\wt\q 1■ sign(Wj)

[0276] As above, [∇ψ(w)]ᵢ and wᵢ are the ith components of the vectors ∇ψ(w) and w, respectively. Note that the above gradient is one-to-many since any weights with absolute value (1 / β)^(1 / (q-1-ε)) can get mapped to an entire interval. In these cases, the inverseof the gradient of the potential may be given by:[∇⁻¹ψ(x)]ᵢ = sign(xᵢ) (25)sign(xj) otherwiseNote that the above inverse gradient is many-to-one if the absolute value of any q-l-ecomponent of x is in the interval (1 + E), it gets mapped to the singlepoint with absolute value1 £

[0277] By using the above (ℓqnorm-directed) equations (24) and (25) with the inverse mapping (of the gradient of the potential) required for the weight update processes of equation (9) for SMD (or equation (14) for RMD), systems in accordance with several embodiments may implement the one-step process for SMD with anorm. By doing the same with equation (10) for SMD (or equation (15) for RMD), systems may implement the two-step process. Nevertheless, in the latter case, as expressed above, the need for equation (24) (i.e., [∇ψ(w)]ᵢ) may be limited only to the time iteration of initialization (e.g., while equation (25) is used at every iteration).ii. Additional Quantization Variations

[0278] Systems and methods in accordance with various embodiments of the invention may utilize variations on and / or alternatives to the above potential and loss function configurations with respect to quantization. A number of alternative potential and loss functions that can be utilized in various embodiments of the invention are discussed further below.1. Rigid Cage Quantization

[0279] In several embodiments of the invention, an alternative convex potential function (the “rigid cage” potential function) may be implemented to obtain bimodal weight convergence. When applied to SMD, this function may follow the formula:ψ(w) = ||w||∞ + (ε / 2)||w||² = max_{i=1,...,p} |w|ᵢ + (ε / 2)∑wᵢ² (26)2 i=i,...,p 22—1i=l for small e > 0 where IlwHo, = maxf lw represents the largest element of w in absolute value. In numerous embodiments, running SMD for >(w) = llwlloo + f ||w||2, when initialized by w0= 0 (and / or where the entries of the weight vector are mostly zero), can result in a final weight vector that is close to bimodal. The rigid cage potential function for RMD similarly may follow the same formula:€ψ(w) = ||w||∞ + (ε / 2)||w||²for q » 1.

[0280] When the potential function follows the above configuration, ψ(w) = ||w||∞ + (ε / 2)||w||²2, the gradient of the potential may be set to:? >(w) = s + ew, where Hsl^ = 1 and s, = 0, if |w,| < || ||oo. (27) where the gradient is a one-to-many mapping. When the absolute value of multiple components of w match HwH^, i.e., when multiple components attain the maximum, then s can be any vector of unit • / ’1norm on the support of components that achieve this maximum.

[0281] In cases where, during initialization of the weights w(0) = w0, there are multiple entries of w₀ that achieve ||w₀||∞, then the gradient ∇ψ(w₀) may be considerednon-unique. Any vector s, with the appropriate signs and unit ℓ₁ norm on the support of the maximum-achieving entries, gives a valid gradient. In various embodiments of the invention s may be chosen uniformly at random, subject to these constraints.

[0282] In some embodiments of the invention, gradient and inverse gradient computations may or may not be done component-wise (e.g., [∇ψ(w)]ᵢ depending only on wᵢ and [∇ψ(x)]ᵢ depending only on xᵢ). In the rigid cage implementation especially, neither mapping may need to be done component-wise. Specifically, the s vector in the gradient may depend on all the entries w attaining ||w||∞. The required sorting may add to the computational complexity of each update step. In such cases, sorting p numbers involves a complexity of (plog p), whereas each gradient update is O(p). This can potentially lead to a slow down of O(log p) in training the model. Fortunately, at time t, the gradient vector ∇ψ(w(t)) = x(t) has already been sorted. Therefore, since the gradient update x(t + 1) = ∇ψ(w(t + 1)) = ∇ψ(w(t)) − η∇ℓ(t, w(t)) = x(t) −only slightly perturbs x(t), it can be expected that the ordering of the components of x(t + 1) may not be significantly different from the ordering of the components of x(t). This can be used to speed up the sorting of x(t + 1) from O(plog p), since x(t) is already sorted. There are many fast algorithms to sort "mostly sorted arrays" that can be applied in a similar fashion in accordance with various embodiments of the invention.

[0283] In various embodiments of the invention, the inverse of the gradient mapping ∇⁻¹ψ(x), for any vector x, can be found in the following fashion:(1) Sort the magnitude of the entries of x in descending order, i.e|x_{i₁}| ≥ |x_{i₂}| ≥ ···, |x_{iₚ}|.(2) Find the largest value of k such that ∑ⱼ₌₁ᵏ |x_{iⱼ}| > k|x_{i_{k+1}}| + 11;.1-1(3) Set α = (∑ⱼ₌₁ᵏ |x_{iⱼ}| − 1) / k; on doing so, the inverse gradient may follow the form:[∇⁻¹ψ(x)]_{iⱼ} = (α / ε)·sign(x_{iⱼ}), j = 1,...,kXi7j = k + l,...,pIn accordance with miscellaneous embodiments of the invention, the inverse of the gradient may follow a many-to-one mapping. In particular, this is because any vector xwhich satisfies step 2 has its k largest components mapped to α / ε (in absolute value). Moreover, as mentioned above, the inverse mapping may not be done component-wise. Specifically, this mapping involves sorting the components of x by absolute value and computing the values of a and k from (potentially) all the entries, adding to the computational complexity of each update step.

[0284] For this reason, the rigid cage SMD (and / or RMD) may be used to quantize weights to one-bit because the inverse gradient mapping in the SMD updates may force the weights to all have the same absolute value. By using the above (rigid cage-directed) equations (27) and (28) with the inverse mapping (of the gradient of the potential) required for the weight update processes of equation (10) for SMD (or equation (15) for RMD), systems in accordance with multiple embodiments may implement the two-step process for updating weights (using mirror descent algorithms with a rigid cage configuration). Systems in accordance with various embodiments of the invention may elect to avoid a one-step version of rigid cage-based mirror descent algorithms. This may be done on the basis that only storing the weights w(t) may, at each time t, cause ambiguity with regards to computing V >(w(t)) forthose components of w(t) that achieve || w(t) || oo. The two-step procedure avoids this because it stores and propagates ^(t) = V^(w(t)) at each iteration and never needs to compute it from the weights. In the two-step case, as expressed above, the need for equation (27) (i.e., [∇⁻¹ψ(x)]_{iⱼ}) may be limited only to the time iteration of initialization (e.g., while equation (28) is used at every iteration). In particular, as referenced above, the fact that equation (28) is many-to-one and maps the top k (in magnitude) components of the gradient to the same value may allow systems in accordance with some embodiments to quantize to a single bit.2. Kurtosis Quantization

[0285] Systems and methods in accordance with numerous embodiments of the invention may, additionally or alternatively, improve the impact of mirror descent on quantization by modifying the loss functions ℓ(·,·). In various embodiments, SMD and / or RMD may be bolstered (e.g., with respect to quantization) by adding Kurtosis terms to the loss functions. For example, for a given network with k layers, where the weights in theith layer of the network are denoted by the vector w^, the Kurtosis loss term may take the form:k Vi / / Vi x \ 4\ \2£ I h“’| ) (29) i=i y=i \ \ J=I / / where ptis the number of weights in the ith layer and X?=iPi = P- When using Kurtosis loss terms, optimization of the loss may then take the form:+ / >4wr(w) (30) where 8 > 0 is a small constant.

[0286] In some embodiments of the invention, Kurtosis loss may be considered alongside various potential functions including but not limited to theqpotential and the rigid cage potential. Systems in accordance with some embodiments may implement the process for SMD (and / or RMD) with various potential functions including but not limited to a norm potential and / or a rigid cage potential. In these cases, the gradient of the loss (for the Kurtosis term) may take the form:[∇ℓ_{kur}(w)]ⱼ^(i) = 8(|wⱼ^(i)|⁴ − (1 / pᵢ)||w^(i)||₁⁴)·(|wⱼ^(i)|³ − (1 / pᵢ⁴)·||wⱼ^(i)||₁³·|w^(i)|)sign(wⱼ^(i)) (31) In such cases, the above gradient (with the weighting β > 0) may be added to the (gradient of the) standard loss to update the gradient of the loss (e.g., that is used to update weight estimates as in equations (9), (10), (14), and / or (15)). As referenced above, this may be done in a one-step and / or two-step process as in the above scenarios.iii. Additional Pruning Variations

[0287] Systems and methods in accordance with some embodiments of the invention may, additionally or alternatively, utilize variations on and / or alternatives to the above potential and loss function configurations with respect to pruning.1. Elastic Net Pruning

[0288] In several embodiments of the invention, an alternative convex potential function (the “elastic net” potential function) may be implemented to encourage derivationof sparse (final) weight vectors. When applied to SMD, this function may follow the formula:ψ(w) = ||w||₁ + (ε / 2)||w||² = ∑(|wᵢ| + (ε / 2)wᵢ²), (32) i=l for small E > 0. In some embodiments, running SMD for ψ(w) = ||w||₁ + (ε / 2)||w||², wheninitialized by w0= 0 (and / or where the entries of the weight vector are mostly zero), will result in a final weight vector that is sparse. The elastic net potential for RMD may follow the same formula:pψ(w) = ||w||₁ + (ε / 2)||w||² = ∑(|wᵢ| + (ε / 2)wᵢ²),Z Z_ I ' Z 'i=lfor small e > 0.

[0289] When the potential is ψ(w) = ||w||₁ + (ε / 2)||w||², the gradient of the potentialmay be set to:(VWW)], = f (i^i+ e)w‘= Sign(w‘) + ew‘if* ° (33)v, |v| < 1 if Wj = 0 where the gradient is a one-to-many mapping (since, when a component of w is zero, the corresponding component of the gradient can be any magnitude no greater than one).

[0290] Additionally or alternatively, the inverse of the gradient may follow the form:[∇⁻¹ψ(x)]ᵢ = (|xᵢ| − 1)(sign(xᵢ) / ε) if |xᵢ| ≥ 1( 0 if |%d < l where the inverse of the gradient is a many-to-one mapping (since, when a component of x has magnitude less than one the corresponding component of the inverse gradient is zero). The above configuration may enable SMD (and / or RMD) implemented using the elastic net to prune weights because the inverse gradient mappings (e.g., equations (9), (10), (14), and (15)) may force some of the weights to be zero.

[0291] By using the above (elastic net-directed) equations (33) and (34) with the inverse mapping (of the gradient of the potential) required for the weight update processes of equation (10) for SMD (or equation (15) for RMD), systems in accordancewith certain embodiments may implement the two-step process for updating weights (via mirror descent algorithms with elastic net configurations). Systems in accordance with various embodiments of the invention may elect to avoid a one-step version of elastic net-based mirror descent algorithms. This may be done on the basis that only storing the weights w(t) may, at each time t, cause ambiguity with regards to computing V^(w(t)) for those components of w(t) that are zero. In the two-step procedure storing and propagating ξ(t) = ∇ψ(w(t)) at each iteration avoids the need to compute it from the weights.

[0292] In accordance with many embodiments of the invention, in implementing the elastic net mirror descent algorithm, when at least one component of the initial weight vector w(0) = w0is set to a non-zero value, then after convergence it may remain zero and preserve the sign of what it was initialized to. Therefore, to maximize the amount of pruning in elastic net mirror descent algorithms, systems in accordance with various embodiments may initialize with all zero weight vector(s). In some cases, as initializing the weights in all zeros may result in zero gradients (meaning that training will not happen) for some neural networks, systems may be configured to initialize with weight vectors with (very few) nonzero elements that are sufficient to result in nonzero gradients and to jumpstart the training process. In doing this, for weights that are initially set to zero, the corresponding components of the gradient can be set to any value in the interval [-1,1]. For example, systems may optimize the process by choosing components of the gradient uniformly.2. Group Pruning

[0293] Systems in accordance with various embodiments of the invention may facilitate simultaneous pruning of entire groups of weights. For example, in a neural network, systems may be configured to remove an entire neuron (and all the weights that are connected to it). In such cases, for a scenario where weights are partitioned into k groups:plw = w®two potentials that may be used, by mirror descent algorithms performed in accordance with multiple embodiments of the invention, to encourage group sparsity may include (but are not limited to):ψ(w) = ∑||w(i)||1+ε; and (35A) ψ(w) = ∑(||w(i)|| + ε / 2||w(i)||2), (35B)i=l for small e > 0. In accordance with some embodiments of the invention, systems may be configured to update weights using the above potential functions.

[0294] When the weight vector is partitioned into k groups:[w«]w —,and the potential ψ(w) = ∑k i=1 ||w(i)||1+ε, systems configured in accordance with multiple embodiments of the invention may determine:w®[∇ψ(w)](i) = (1 + ε) w(i) / ||w(i)||1-ε, (36) where w® is the weight vector of the i th group and [V >(w)]® is the corresponding weight vector of the gradient of the potential. Additionally or alternatively, the inverse of the gradient of the potential may be given by:[∇-1ψ(x)](i) = ||x(i)||(1-ε) / ε / (1+ε)1 / ε x(i). (37)

[0295] By using the above (ℓq norm-directed) equations (36) and (37) with the inverse mapping (of the gradient of the potential) required for the weight update processes of equation (9) for SMD (or equation (14) for RMD), systems in accordance with some embodiments may implement the one-step process for mirror descent with a ℓq norm. By doing the same with equation (10) for SMD (or equation (15) for RMD), systems may implement the two-step process. Nevertheless, in the latter case, asexpressed above, the need for equation (36) (i.e., [∇ψ(w)]i) may be limited to the time iteration of initialization (e.g., while equation (37) is used at every iteration).

[0296] When the weight vector is partitioned into k groups:[w«lw =,and the potential ψ(w) = ∑k i=1 (||w(i)|| + ε / 2||w(i)||2), systems configured in accordance with several embodiments of the invention may determine:[(1 / ||w(i)|| + ε) w(i) if w(i) ≠ 0 (38)v, ||v|| ≤ 1 if w(i) = 0 where w(i) is the weight vector of the i th group and [∇ψ(w)](i) is the corresponding weight vector of the gradient of the potential. In these cases, the gradient is a one-to-many mapping, since if a group component of w is the all zero vector then the corresponding group component of the gradient can be any vector of of norm no greater than one.

[0297] Additionally or alternatively, the inverse of the gradient of the potential may be given by:7l llrx®l lll - 1x(o jf || (OH > i[V ^(x)]® = E|| ® || I' II. (39)( 0 if ||x®|| < 1In these cases, the inverse of the gradient is a many-to-one mapping, since when a group component of x has a norm that is less than one, the corresponding group component of the inverse gradient may be an all zero vector. Using the above formulae, the elastic net SMD may simultaneously prune all the weights in a group through mirror descent algorithms on the basis that the inverse gradient mapping forces some of the group weights to be the all zero vector.

[0298] In accordance with several embodiments of the invention, elastic net potential functions may be used alongside attempts at group pruning (“group pruning elastic net” configurations). In such cases, by using the above (elastic net-directed) equations (38) and (39) with the inverse mapping (of the gradient of the potential) required for the weight update processes of equation (10) for SMD (or equation (15) for RMD),systems in accordance with various embodiments may implement the two-step process for updating weights (using mirror descent algorithms with group pruning elastic net configurations). Systems in accordance with many embodiments of the invention may elect to avoid a one-step version of group pruning elastic net-based mirror descent algorithms. This may be done on the basis that only storing the weights w(t) may, at each time t, cause ambiguity with regards to computing V^(w(t)) for those components of w(t) that are zero vectors. In the two-step procedure storing and propagating (t) = V^(w(t)) at each iteration avoids the need to compute it from the weights.

[0299] In accordance with many embodiments of the invention, in implementing the elastic net mirror descent algorithm, when at least one component of the initial weight vector w(0) = w0is set to a non-zero value, then after convergence it may remain zero and preserve the direction of what it was initialized to. Therefore, to maximize the amount of pruning in elastic net mirror descent algorithms, systems in accordance with miscellaneous embodiments may initialize every group with the all zero weight vector. As initializing the weights as all zeros may result in zero gradients (meaning that training will not happen) for some neural networks, systems may be configured to initialize with weight vectors with very few (groups of) nonzero elements that are sufficient to result in nonzero gradients and to jumpstart the training process. In doing this, for weights that are initially set to zero, the corresponding components of the gradient can be set to any value whose norm is < 1. For example, systems may optimize the process by choosing components of the gradient uniformly from within a unit ball.

[0300] Systems and methods in accordance with various embodiments of the invention are not limited to the above compression objectives. Accordingly, it should be appreciated that the capabilities of SMD and / or RMD algorithms described above with respect to FIGS. 5-7 can also be implemented outside the context of quantization and / or pruning. As can readily be appreciated, the specific objectives pursued during training-concurrent compression processes implemented in accordance with various embodiments of the invention are largely dependent upon the requirements of specific applications.III. Model TrainingA. Training System

[0301] An example of a training system that trains models in accordance with miscellaneous embodiments of the invention is illustrated in FIG. 8. The training system 800 includes one or more server systems such as (but not limited to) point of presence (PoP) servers 810, inference servers 840, and / or data center servers 870 that are connected via a communications network 860. In many embodiments, one or more data center servers 870 gather data and train one or models using a training-concurrent compression process such as (but not limited to) any of the various training-concurrent compression processes described above. The trained models can then be deployed to one or more inference server clusters that can be distributed geographically to enable low-latency responses to requests from user devices. In a number of embodiments, point of presence servers 810 communicate with user devices and route requests to a proximate inference server capable of responding to a request using trained machine learning models. In many embodiments, multiple trained models are deployed within the network and the point of presence servers route requests to inference servers that have the capability to generate responses using a particular trained model relevant to the request. In a number of embodiments, the server systems utilized in one or more of the PoP servers 810, inference servers 840, and / or data center servers 870 can be implemented using the NVIDIA HGX AI platform manufactured by Nvidia Corp, of Sunnyvale, California.

[0302] The communications network 860 can be a network such as the Internet that allows devices connected to the network 860 to communicate with other connected devices. Server systems 810, 840, and 870 are connected to the network 860. Each of the server systems 810, 840, and 870 is a group of one or more servers communicatively connected to one another via internal networks that execute processes that provide cloud services to users over the network 860. One skilled in the art will recognize that a training system may exclude certain components and / or include other components that are omitted for brevity without departing from this invention. As can readily be appreciated,the specific architectures of given training systems largely depend upon the requirements of specific applications.

[0303] For purposes of this discussion, cloud services are one or more applications that are executed by one or more server systems to provide data and / or executable applications to devices over a network. The server systems 810, 840, and 870 are shown each having three servers in the internal network. However, the server systems 810, 840 and 870 may include any number of servers and any additional number of server systems may be connected to the network 860 to provide cloud services. In accordance with various embodiments of this invention, a training system that uses systems and methods that train and / or utilize models in accordance with an embodiment of the invention may be provided by a process being executed on a single server system and / or a group of server systems communicating over network 860.

[0304] Users may use personal devices 880 and 820 that connect to the network 860 to perform processes that train and / or utilize models in accordance with various embodiments of the invention. In the shown embodiment, the personal devices 880 are shown as desktop computers that are connected via a conventional “wired” connection to the network 860. However, the personal device 880 may be a desktop computer, a laptop computer, a smart television, an entertainment gaming console, or any other device that connects to the network 860 via a “wired” connection. The mobile device 820 connects to network 860 using a wireless connection. A wireless connection is a connection that uses Radio Frequency (RF) signals, Infrared signals, or any other form of wireless signaling to connect to the network 860. In the example of this figure, the mobile device 820 is a mobile telephone. However, mobile device 820 may be a mobile phone, Personal Digital Assistant (PDA), a tablet, a smartphone, or any other type of device that connects to network 860 via wireless connection without departing from this invention.

[0305] As can readily be appreciated the specific computing system used to train models is largely dependent upon the requirements of a given application and should not be considered as limited to any specific computing system(s) implementation.B. Training Element

[0306] An example of a training element that executes instructions to perform processes that train and / or utilize models in accordance with various embodiments of the invention is illustrated in FIG. 9. Training elements in accordance with many embodiments of the invention can include (but are not limited to) one or more of mobile devices, cameras, and / or computers. Training element 900 includes processor 905, peripherals 910, network interface 915, and memory 920. One skilled in the art will recognize that a training element 900 may exclude certain components and / or include other components that are omitted for brevity without departing from this invention.

[0307] The processor 905 can include (but is not limited to) a processor, microprocessor, controller, or a combination of processors, microprocessor, and / or controllers that performs instructions stored in the memory 920 to manipulate data stored in the memory. Processor instructions can configure the processor 905 to perform processes in accordance with certain embodiments of the invention.

[0308] Peripherals 910 can include any of a variety of components for capturing data, such as (but not limited to) cameras, displays, and / or sensors. In a variety of embodiments, peripherals 910 can be used to gather inputs and / or provide outputs. Training elements 900 can utilize network interfaces 915 to transmit and receive data over a network based upon the instructions performed by processor 905. Peripherals 910 and / or network interfaces 915 in accordance with many embodiments of the invention can be used to gather inputs that can be used to train models operating in accordance with many embodiments of the invention.

[0309] Memory 920 includes a training application 925, training data 930, and model data 935. Training applications in accordance with several embodiments of the invention can be used to train models using potential functions, weight vectors, and / or auxiliary variables.

[0310] Training data 930 in accordance with many embodiments of the invention can include various types of training data 930 (or samples), such as (but not limited to) video, audio, text, images, etc. In various embodiments, training data 930 may include labels for the training data. Training data 930 in accordance with some embodiments of theinvention can be received continuously, where training applications can update the model continuously as new data is received.

[0311] In several embodiments, model data can store various parameters, auxiliary variables, and / or weights for models. Model data in accordance with many embodiments of the invention can be updated through training on training data 930 captured on a training element 900 or can be trained remotely and updated at a training element 900. In a variety of embodiments, model data can include data for a pre-trained model that can be updated based on a new set of training data 930.

[0312] Although a specific example of a training element 900 is illustrated in this figure, any of a variety of training elements can be utilized to perform processes for training models similar to those described herein as appropriate to the requirements of specific applications in accordance with embodiments of the invention.C. Training Application

[0313] An example of a training application for training models in accordance with some embodiments of the invention is illustrated in FIG. 10. Training application 1000 includes potential selection engine 1005, loss computation engine 1010, update engine 1015, and output engine 1020. One skilled in the art will recognize that a training application may exclude certain components and / or include other components that are omitted for brevity without departing from this invention.

[0314] Potential selection engines 1005 in accordance with numerous embodiments of the invention can select potential functions to be used in training models. In numerous embodiments, potentials can be selected based on desired characteristics of the output model (e.g., generalization, sparsity, etc.).

[0315] In a number of embodiments, loss computation engines 1010 can compute losses in accordance with various methods described throughout this specification. In doing so, loss computation engines 1010 may be configured to compute loss components and regularizing components. In some embodiments, loss components can include but are not limited to constraint-enforcing loss. Constraint-enforcing losses in accordance with some embodiments of the invention can be computed based on auxiliary variablesassociated with each element of a training dataset. In a variety of embodiments, auxiliary variables are only used in training of the model and are not part of the output model.

[0316] Regularizing components of loss functions implemented in accordance with many embodiments of the invention can be computed by applying a potential function to weights of a model. Potential functions in accordance with a variety of embodiments of the invention can include but are not limited to various -norm potentials, where q is a number (e.g., 1, 2, 3, 10, etc.), and / or a negative entropy potential. Regularizing components in accordance with certain embodiments of the invention can be selected to optimize closeness to the initialized model, where closeness can be computed as a Bregman divergence.

[0317] Update engines 1015 in accordance with certain embodiments of the invention can update weights of models and / or auxiliary variables throughout an optimization process. In a number of embodiments, update engines 1015 can update weights based on computed losses as described herein. Update engines 1015 in accordance with several embodiments of the invention can update auxiliary variables based on gradients for constraint-enforcing losses.

[0318] In a variety of embodiments, output engines 1020 can provide a variety of outputs to a user, including (but not limited to) weights and / or outputs for models. Outputs for a model in accordance with a variety of embodiments of the invention can include (but are not limited to) classifications, regressions, clusters, etc. In certain embodiments, outputs can include computed losses for a subset of a dataset, where another training application can update weights of a model based on losses computed at multiple different processors.

[0319] Although a specific example of a training application is illustrated in this figure, any of a variety of training applications can be utilized to perform processes for training models similar to those described herein as appropriate to the requirements of specific applications in accordance with embodiments of the invention.IV. Exemplary Embodiments

[0320] Systems and methods for training-concurrent compression of neural network models are illustrated. A first embodiment, comprising a method for training-concurrent pruning, performed by one or more processors. The method obtains a pretrained neural network model, wherein the pretrained neural network model comprises initial values for a potential function and a set of weights. The method trains a neural network model based upon the pretrained neural network model and a set of training samples, wherein training the neural network model comprises, for each of a plurality of time iterations: calculating loss for a current neural network model having a current set of weights at a current time iteration using a loss function; updating a gradient of the potential function at the current time iteration according to a gradient of the loss; updating an inverse mapping of the gradient of the potential function; creating an updated neural network model including an updated set of weights for a next time iteration according to the gradient of the potential function and the inverse mapping of the gradient of the potential function, wherein the updated set of weights is obtained by pruning at least a portion of the current set of weights of the current neural network model; and storing the updated set of weights.

[0321] A second embodiment, including the features of the first embodiment and further comprising the trained neural network model is a multi-layered neural network model.

[0322] A third embodiment, including the features of the second embodiment and further comprising the multi-layered neural network model is selected from the group consisting of a convolutional neural network model, a recursive neural network model, a multi-layer perceptron, and a generative adversarial network.

[0323] A fourth embodiment, including the features of the second embodiment and further comprising the multi-layered neural network model is a large language model.

[0324] A fifth embodiment, including the features of any of the first to fourth embodiments and further comprising the updated set of weights is further obtained based on at least one additional vector comprising a current set of auxiliary variables for the current neural network model.

[0325] A sixth embodiment, including the features of the fifth embodiment and further comprising each sample of the set of training samples corresponds to a different auxiliary variable of the current set of auxiliary variables.

[0326] A seventh embodiment, including the features of the fifth or sixth embodiment and further comprising obtaining the updated set of weights for the next time iteration comprises obtaining an updated set of auxiliary variables based on the current set of weights for the current time iteration.

[0327] An eighth embodiment, including the features of the seventh embodiment and further comprising each auxiliary variable of the current set of auxiliary variables is obtained based on a regularization parameter and the loss function evaluated for the current set of weights at the current time iteration.

[0328] A ninth embodiment, including the features of the seventh or eighth embodiment and further comprising a regularizer mirror descent (RMD) process is used to obtain at least one selected from the group consisting of the updated set of weights; and the current set of auxiliary variables.

[0329] A tenth embodiment, including the features of any of the seventh to ninth embodiments and further comprising iterative updates are used to obtain at least one selected from the group consisting of the updated set of weights; and the updated set of auxiliary variables.

[0330] An eleventh embodiment, including the features of any of the fifth to tenth embodiments and further comprising each of the current set of weights and the current set of auxiliary variables is initialized to at least one selected from the group consisting of: a zero vector and a non-zero vector comprising random numbers within a predetermined threshold of zero.

[0331] A twelfth embodiment, including the features of any of the fifth to eleventh embodiments and further comprising the loss function comprises a constraint-enforcing loss computed from the current set of auxiliary variables.

[0332] A thirteenth embodiment, including the features of the twelfth embodiment and further comprising the constraint-enforcing loss is applied to prevent overfitting the neural network model to the set of training samples.

[0333] A fourteenth embodiment, including the features of any of the first to fourth embodiments and further comprising a stochastic mirror descent (SMD) process is used to obtain the updated set of weights.

[0334] A fifteenth embodiment, including the features of any of the first to fourteenth embodiments and further comprising the loss function includes a Kurtosis term.

[0335] A sixteenth embodiment, including the features of the fifteenth embodiment and further comprising the Kurtosis term is a function of the current set of weights.

[0336] A seventeenth embodiment, including the features of any of the first to sixteenth embodiments and further comprising supervised learning is used to train the neural network model.

[0337] An eighteenth embodiment, including the features of the seventeenth embodiment and further comprising the loss function is configured to only equal zero when a predicted output value of the neural network model exactly matches a labeled output value from the set of training samples.

[0338] A nineteenth embodiment, including the features of any of the first to eighteenth embodiments and further comprising the loss function is selected from the group consisting of a quadratic loss function and a cross-entropy loss function.

[0339] A twentieth embodiment, including the features of any of the first to nineteenth embodiments and further comprising, at the current time iteration, obtaining the updated set of weights is based, at least in part, on a learning rate at the current time iteration.

[0340] A twenty-first embodiment, including the features of the twentieth embodiment and further comprising the learning rate is time-invariant.

[0341] A twenty-second embodiment, including the features of any of the first to twenty-first embodiments and further comprising updating the gradient of the potential function at the current time iteration is performed using at least one system selected from the group consisting of Adam, AdamW, AdaGrad, RMSProp, Momentum, and WeightDecay.

[0342] A twenty-third embodiment, including the features of any of the first to twenty-first embodiments and further comprising the gradient of the potential function at the current time iteration is obtained recursively based on the gradient of the potential function at a previous time iteration.

[0343] A twenty-fourth embodiment, including the features of the twenty-third embodiment and further comprising training the neural network model at the current timeiteration further comprises storing the gradient of the potential function at the current time iteration.

[0344] A twenty-fifth embodiment, including the features of any of the first to twenty-fourth embodiments and further comprising, after the plurality of time iterations, the potential function is minimized via implicit regularization.

[0345] A twenty-sixth embodiment, including the features of any of the first to twenty-fifth embodiments and further comprising the potential function is a strictly convex function.

[0346] A twenty-seventh embodiment, including the features of the twenty-sixth embodiment and further comprising the potential function is a norm function.

[0347] A twenty-eighth embodiment, including the features of the twenty-sixth or twenty-seventh embodiment and further comprising the potential function is selected from the group consisting of: an Lqnorm function where q » 1; and an £1+enorm function where 1 » e > 0.

[0348] A twenty-ninth embodiment, including the features of any of the first to twenty-eighth embodiments and further comprising training the neural network model comprises at least one selected from the group consisting of distributed training, batch training, synchronous training, ensemble learning, meta learning, transfer learning, and continual learning.

[0349] A thirtieth embodiment, including the features of any of the first to twentyninth embodiments and further comprising the inverse mapping of the gradient of the potential function is a many-to-one function.

[0350] A thirty-first embodiment, including the features of any of the first to thirtieth embodiments and further comprising obtaining the updated set of weights bimodally concentrates a distribution of the set of weights.

[0351] A thirty-second embodiment, including the features of the thirty-first embodiment and further comprising quantizing the neural network model after the plurality of time iterations is performed according to a bimodality of the distribution of the set of weights.

[0352] A thirty-third embodiment, including the features of the thirty-first or thirty-second embodiment and further comprising a data size of each individual weight of the current set of weights is quantized to a single bit.

[0353] A thirty-fourth embodiment, including the features of any of the first to thirty-third embodiments and further comprising the potential function is selected to induce sparsity within the set of weights.

[0354] A thirty-fifth embodiment, including the features of the thirty-fourth embodiment and further comprising the potential function is selected to induce group sparsity within a trained set of weights for the trained neural network model.

[0355] A thirty-sixth embodiment, including the features of the thirty-fourth or thirtyfifth embodiment and further comprising the potential function is an elastic net.

[0356] A thirty-seventh embodiment comprising a method for training-concurrent quantization, performed by one or more processors. The method obtains a pretrained neural network model, wherein the pretrained neural network model comprises initial values for a potential function and a set of weights. The method trains a neural network model based upon the pretrained neural network model and a set of training samples, wherein training the neural network model comprises, for each of a plurality of time iterations: calculating loss for a current neural network model having a current set of weights at a current time iteration using a loss function; updating a gradient of the potential function at the current time iteration according to a gradient of the loss; updating an inverse mapping of the gradient of the potential function; creating an updated neural network model including an updated set of weights for a next time iteration according to the gradient of the potential function and the inverse mapping of the gradient of the potential function, wherein creating the updated neural network model bimodally concentrates a distribution of the updated set of weights; and storing the updated set of weights. The method quantizes the trained neural network model after the plurality of time iterations, wherein quantizing the trained neural network model is performed according to a bimodality of the distribution of a trained set of weights.

[0357] A thirty-eighth embodiment, including the features of the thirty-seventh embodiment and further comprising the trained neural network model is a multi-layered neural network model.

[0358] A thirty-ninth embodiment, including the features of the thirty-eighth embodiment and further comprising the multi-layered neural network model is selected from the group consisting of a convolutional neural network model, a recursive neural network model, a multi-layer perceptron, and a generative adversarial network.

[0359] A fortieth embodiment, including the features of the thirty-eighth embodiment and further comprising the multi-layered neural network model is a large language model.

[0360] A forty-first embodiment, including the features of any of the thirty-seventh to fortieth embodiments and further comprising the updated set of weights is further obtained based on at least one additional vector comprising a current set of auxiliary variables for the current neural network model.

[0361] A forty-second embodiment, including the features of the forty-first embodiment and further comprising each sample of the set of training samples corresponds to a different auxiliary variable of the current set of auxiliary variables.

[0362] A forty-third embodiment, including the features of the forty-first or forty-second embodiment and further comprising obtaining the updated set of weights for the next time iteration comprises obtaining an updated set of auxiliary variables based on the current set of weights for the current time iteration.

[0363] A forty-fourth embodiment, including the features of the forty-third embodiment and further comprising each auxiliary variable of the current set of auxiliary variables is obtained based on a regularization parameter and the loss function evaluated for the current set of weights at the current time iteration.

[0364] A forty-fifth embodiment, including the features of the forty-third or fortyfourth embodiment and further comprising a regularizer mirror descent (RMD) process is used to obtain at least one selected from the group consisting of the updated set of weights; and the current set of auxiliary variables.

[0365] A forty-sixth embodiment, including the features of any of the forty-third to forty-fifth embodiments and further comprising iterative updates are used to obtain at least one selected from the group consisting of the updated set of weights; and the updated set of auxiliary variables.

[0366] A forty-seventh embodiment, including the features of any of the forty-first to forty-sixth embodiments and further comprising each of the current set of weights and the current set of auxiliary variables is initialized to at least one selected from the group consisting of: a zero vector and a non-zero vector comprising random numbers within a predetermined threshold of zero.

[0367] A forty-eighth embodiment, including the features of any of the forty-first to forty-seventh embodiments and further comprising the loss function comprises a constraint-enforcing loss computed from the current set of auxiliary variables.

[0368] A forty-ninth embodiment, including the features of the forty-eighth embodiment and further comprising the constraint-enforcing loss is applied to prevent overfitting the neural network model to the set of training samples.

[0369] A fiftieth embodiment, including the features of any of the thirty-seventh to fortieth embodiments and further comprising a stochastic mirror descent (SMD) process is used to obtain the updated set of weights.

[0370] A fifty-first embodiment, including the features of any of the thirty-seventh to fiftieth embodiments and further comprising the loss function includes a Kurtosis term.

[0371] A fifty-second embodiment, including the features of the fifty-first embodiment and further comprising the Kurtosis term is a function of the current set of weights.

[0372] A fifty-third embodiment, including the features of any of the thirty-seventh to fifty-second embodiments and further comprising supervised learning is used to train the neural network model.

[0373] A fifty-fourth embodiment, including the features of the fifty-third embodiment and further comprising the loss function is configured to only equal zero when a predicted output value of the neural network model exactly matches a labeled output value from the set of training samples.

[0374] A fifty-fifth embodiment, including the features of any of the thirty-seventh to fifty-fourth embodiments and further comprising the loss function is selected from the group consisting of a quadratic loss function and a cross-entropy loss function.

[0375] A fifty-sixth embodiment, including the features of any of the thirty-seventh to fifty-fifth embodiments and further comprising, at the current time iteration, obtainingthe updated set of weights is based, at least in part, on a learning rate at the current time iteration.

[0376] A fifty-seventh embodiment, including the features of the fifty-sixth embodiment and further comprising the learning rate is time-invariant.

[0377] A fifty-eighth embodiment, including the features of any of the thirty-seventh to fifty-seventh embodiments and further comprising updating the gradient of the potential function at the current time iteration is performed using at least one system selected from the group consisting of Adam, AdamW, AdaGrad, RMSProp, Momentum, and WeightDecay.

[0378] A fifty-ninth embodiment, including the features of any of the thirty-seventh to fifty-seventh embodiments and further comprising the gradient of the potential function at the current time iteration is obtained recursively based on the gradient of the potential function at a previous time iteration.

[0379] A sixtieth embodiment, including the features of the fifty-ninth embodiment and further comprising training the neural network model at the current time iteration further comprises storing the gradient of the potential function at the current time iteration.

[0380] A sixty-first embodiment, including the features of any of the thirty-seventh to sixtieth embodiments and further comprising, after the plurality of time iterations, the potential function is minimized via implicit regularization.

[0381] A sixty-second embodiment, including the features of any of the thirtyseventh to sixty-first embodiments and further comprising the potential function is a strictly convex function.

[0382] A sixty-third embodiment, including the features of the sixty-second embodiment and further comprising the potential function is a norm function.

[0383] A sixty-fourth embodiment, including the features of the sixty-second or sixty-third embodiment and further comprising the potential function is selected from the group consisting of: an £qnorm function where q » 1; and an £1+snorm function where 1 » E > 0.

[0384] A sixty-fifth embodiment, including the features of any of the thirty-seventh to sixty-fourth embodiments and further comprising training the neural network model comprises at least one selected from the group consisting of distributed training, batchtraining, synchronous training, ensemble learning, meta learning, transfer learning, and continual learning.

[0385] A sixty-sixth embodiment, including the features of any of the thirty-seventh to sixty-fifth embodiments and further comprising the inverse mapping of the gradient of the potential function is a many-to-one function.

[0386] A sixty-seventh embodiment, including the features of any of the thirtyseventh to sixty-sixth embodiments and further comprising a data size of each individual weight of the current set of weights is quantized to a single bit.

[0387] A sixty-eighth embodiment, including the features of any of the thirtyseventh to sixty-seventh embodiments and further comprising the potential function is selected to enable the bimodality of the distribution of the trained set of weights.

[0388] A sixty-ninth embodiment, including the features of the sixty-eighth embodiment and further comprising the potential function is a rigid cage.

[0389] A seventieth embodiment comprising a non-transitory machine-readable medium storing instructions that, when executed by a processor, are configured to cause the processor to perform operations for training-concurrent pruning. The processor obtains a pretrained neural network model, wherein the pretrained neural network model comprises initial values for a potential function and a set of weights. The processor trains a neural network model based upon the pretrained neural network model and a set of training samples, wherein training the neural network model comprises, for each of a plurality of time iterations: calculating loss for a current neural network model having a current set of weights at a current time iteration using a loss function; updating a gradient of the potential function at the current time iteration according to a gradient of the loss; updating an inverse mapping of the gradient of the potential function; creating an updated neural network model including an updated set of weights for a next time iteration according to the gradient of the potential function and the inverse mapping of the gradient of the potential function, wherein the updated set of weights is obtained by pruning at least a portion of the current set of weights of the current neural network model; and storing the updated set of weights.

[0390] A seventy-first embodiment, including the features of the seventieth embodiment and further comprising the trained neural network model is a multi-layered neural network model.

[0391] A seventy-second embodiment, including the features of the seventy-first embodiment and further comprising the multi-layered neural network model is selected from the group consisting of a convolutional neural network model, a recursive neural network model, a multi-layer perceptron, and a generative adversarial network.

[0392] A seventy-third embodiment, including the features of the seventy-first embodiment and further comprising the multi-layered neural network model is a large language model.

[0393] A seventy-fourth embodiment, including the features of any of the seventieth to seventy-third embodiments and further comprising the updated set of weights is further obtained based on at least one additional vector comprising a current set of auxiliary variables for the current neural network model.

[0394] A seventy-fifth embodiment, including the features of the seventy-fourth embodiment and further comprising each sample of the set of training samples corresponds to a different auxiliary variable of the current set of auxiliary variables.

[0395] A seventy-sixth embodiment, including the features of the seventy-fourth or seventy-fifth embodiment and further comprising obtaining the updated set of weights for the next time iteration comprises obtaining an updated set of auxiliary variables based on the current set of weights for the current time iteration.

[0396] A seventy-seventh embodiment, including the features of the seventy-sixth embodiment and further comprising each auxiliary variable of the current set of auxiliary variables is obtained based on a regularization parameter and the loss function evaluated for the current set of weights at the current time iteration.

[0397] A seventy-eighth embodiment, including the features of the seventy-sixth or seventy-seventh embodiment and further comprising a regularizer mirror descent (RMD) process is used to obtain at least one selected from the group consisting of the updated set of weights; and the current set of auxiliary variables.

[0398] A seventy-ninth embodiment, including the features of any of the seventysixth to seventy-eighth embodiments and further comprising iterative updates are used toobtain at least one selected from the group consisting of the updated set of weights; and the updated set of auxiliary variables.

[0399] An eightieth embodiment, including the features of any of the seventy-fourth to seventy-ninth embodiments and further comprising each of the current set of weights and the current set of auxiliary variables is initialized to at least one selected from the group consisting of: a zero vector and a non-zero vector comprising random numbers within a predetermined threshold of zero.

[0400] An eighty-first embodiment, including the features of any of the seventyfourth to eightieth embodiments and further comprising the loss function comprises a constraint-enforcing loss computed from the current set of auxiliary variables.

[0401] An eighty-second embodiment, including the features of the eighty-first embodiment and further comprising the constraint-enforcing loss is applied to prevent overfitting the neural network model to the set of training samples.

[0402] An eighty-third embodiment, including the features of any of the seventieth to seventy-third embodiments and further comprising a stochastic mirror descent (SMD) process is used to obtain the updated set of weights.

[0403] An eighty-fourth embodiment, including the features of any of the seventieth to eighty-third embodiments and further comprising the loss function includes a Kurtosis term.

[0404] An eighty-fifth embodiment, including the features of the eighty-fourth embodiment and further comprising the Kurtosis term is a function of the current set of weights.

[0405] An eighty-sixth embodiment, including the features of any of the seventieth to eighty-fifth embodiments and further comprising supervised learning is used to train the neural network model.

[0406] An eighty-seventh embodiment, including the features of the eighty-sixth embodiment and further comprising the loss function is configured to only equal zero when a predicted output value of the neural network model exactly matches a labeled output value from the set of training samples.

[0407] An eighty-eighth embodiment, including the features of any of the seventieth to eighty-seventh embodiments and further comprising the loss function is selected from the group consisting of a quadratic loss function and a cross-entropy loss function.

[0408] An eighty-ninth embodiment, including the features of any of the seventieth to eighty-eighth embodiments and further comprising, at the current time iteration, obtaining the updated set of weights is based, at least in part, on a learning rate at the current time iteration.

[0409] A ninetieth embodiment, including the features of the eighty-ninth embodiment and further comprising the learning rate is time-invariant.

[0410] A ninety-first embodiment, including the features of any of the seventieth to ninetieth embodiments and further comprising updating the gradient of the potential function at the current time iteration is performed using at least one system selected from the group consisting of Adam, AdamW, AdaGrad, RMSProp, Momentum, and WeightDecay.

[0411] A ninety-second embodiment, including the features of any of the seventieth to ninetieth embodiments and further comprising the gradient of the potential function at the current time iteration is obtained recursively based on the gradient of the potential function at a previous time iteration.

[0412] A ninety-third embodiment, including the features of the ninety-second embodiment and further comprising training the neural network model at the current time iteration further comprises storing the gradient of the potential function at the current time iteration.

[0413] A ninety-fourth embodiment, including the features of any of the seventieth to ninety-third embodiments and further comprising, after the plurality of time iterations, the potential function is minimized via implicit regularization.

[0414] A ninety-fifth embodiment, including the features of any of the seventieth to ninety-fourth embodiments and further comprising the potential function is a strictly convex function.

[0415] A ninety-sixth embodiment, including the features of the ninety-fifth embodiment and further comprising the potential function is a norm function.

[0416] A ninety-seventh embodiment, including the features of the ninety-fifth or ninety-sixth embodiment and further comprising the potential function is selected from the group consisting of: an £qnorm function where q » 1; and an £1+snorm function where 1 » £ > 0.

[0417] A ninety-eighth embodiment, including the features of any of the seventieth to ninety-seventh embodiments and further comprising training the neural network model comprises at least one selected from the group consisting of distributed training, batch training, synchronous training, ensemble learning, meta learning, transfer learning, and continual learning.

[0418] A ninety-ninth embodiment, including the features of any of the seventieth to ninety-eighth embodiments and further comprising the inverse mapping of the gradient of the potential function is a many-to-one function.

[0419] A one hundredth embodiment, including the features of any of the seventieth to ninety-ninth embodiments and further comprising obtaining the updated set of weights bimodally concentrates a distribution of the set of weights.

[0420] A one hundred first embodiment, including the features of the one hundredth embodiment and further comprising quantizing the neural network model after the plurality of time iterations is performed according to a bimodality of the distribution of the set of weights.

[0421] A one hundred second embodiment, including the features of the one hundredth or one hundred first embodiment and further comprising a data size of each individual weight of the current set of weights is quantized to a single bit.

[0422] A one hundred third embodiment, including the features of any of the seventieth to one hundred second embodiments and further comprising the potential function is selected to induce sparsity within the set of weights.

[0423] A one hundred fourth embodiment, including the features of the one hundred third embodiment and further comprising the potential function is selected to induce group sparsity within a trained set of weights for the trained neural network model.

[0424] A one hundred fifth embodiment, including the features of the one hundred third or one hundred fourth embodiment and further comprising the potential function is an elastic net.

[0425] A one hundred sixth embodiment comprising a non-transitory machine-readable medium storing instructions that, when executed by a processor, are configured to cause the processor to perform operations for training-concurrent quantization. The processor obtains a pretrained neural network model, wherein the pretrained neural network model comprises initial values for a potential function and a set of weights. The processor trains a neural network model based upon the pretrained neural network model and a set of training samples, wherein training the neural network model comprises, for each of a plurality of time iterations: calculating loss for a current neural network model having a current set of weights at a current time iteration using a loss function; updating a gradient of the potential function at the current time iteration according to a gradient of the loss; updating an inverse mapping of the gradient of the potential function; creating an updated neural network model including an updated set of weights for a next time iteration according to the gradient of the potential function and the inverse mapping of the gradient of the potential function, wherein creating the updated neural network model bimodally concentrates a distribution of the updated set of weights; and storing the updated set of weights. The processor quantizes the trained neural network model after the plurality of time iterations, wherein quantizing the trained neural network model is performed according to a bimodality of the distribution of a trained set of weights.

[0426] A one hundred seventh embodiment, including the features of the one hundred sixth embodiment and further comprising the trained neural network model is a multi-layered neural network model.

[0427] A one hundred eighth embodiment, including the features of the one hundred seventh embodiment and further comprising the multi-layered neural network model is selected from the group consisting of a convolutional neural network model, a recursive neural network model, a multi-layer perceptron, and a generative adversarial network.

[0428] A one hundred ninth embodiment, including the features of the one hundred seventh embodiment and further comprising the multi-layered neural network model is a large language model.

[0429] A one hundred tenth embodiment, including the features of any of the one hundred sixth to one hundred ninth embodiments and further comprising the updated setof weights is further obtained based on at least one additional vector comprising a current set of auxiliary variables for the current neural network model.

[0430] A one hundred eleventh embodiment, including the features of the one hundred tenth embodiment and further comprising each sample of the set of training samples corresponds to a different auxiliary variable of the current set of auxiliary variables.

[0431] A one hundred twelfth embodiment, including the features of the one hundred tenth or one hundred eleventh embodiment and further comprising obtaining the updated set of weights for the next time iteration comprises obtaining an updated set of auxiliary variables based on the current set of weights for the current time iteration.

[0432] A one hundred thirteenth embodiment, including the features of the one hundred twelfth embodiment and further comprising each auxiliary variable of the current set of auxiliary variables is obtained based on a regularization parameter and the loss function evaluated for the current set of weights at the current time iteration.

[0433] A one hundred fourteenth embodiment, including the features of the one hundred twelfth or one hundred thirteenth embodiment and further comprising a regularizer mirror descent (RMD) process is used to obtain at least one selected from the group consisting of the updated set of weights; and the current set of auxiliary variables.

[0434] A one hundred fifteenth embodiment, including the features of any of the one hundred twelfth to one hundred fourteenth embodiments and further comprising iterative updates are used to obtain at least one selected from the group consisting of the updated set of weights; and the updated set of auxiliary variables.

[0435] A one hundred sixteenth embodiment, including the features of any of the one hundred tenth to one hundred fifteenth embodiments and further comprising each of the current set of weights and the current set of auxiliary variables is initialized to at least one selected from the group consisting of: a zero vector and a non-zero vector comprising random numbers within a predetermined threshold of zero.

[0436] A one hundred seventeenth embodiment, including the features of any of the one hundred tenth to one hundred sixteenth embodiments and further comprising the loss function comprises a constraint-enforcing loss computed from the current set of auxiliary variables.

[0437] A one hundred eighteenth embodiment, including the features of the one hundred seventeenth embodiment and further comprising the constraint-enforcing loss is applied to prevent overfitting the neural network model to the set of training samples.

[0438] A one hundred nineteenth embodiment, including the features of any of the one hundred sixth to one hundred ninth embodiments and further comprising a stochastic mirror descent (SMD) process is used to obtain the updated set of weights.

[0439] A one hundred twentieth embodiment, including the features of any of the one hundred sixth to one hundred nineteenth embodiments and further comprising the loss function includes a Kurtosis term.

[0440] A one hundred twenty-first embodiment, including the features of the one hundred twentieth embodiment and further comprising the Kurtosis term is a function of the current set of weights.

[0441] A one hundred twenty-second embodiment, including the features of any of the one hundred sixth to one hundred twenty-first embodiments and further comprising supervised learning is used to train the neural network model.

[0442] A one hundred twenty-third embodiment, including the features of the one hundred twenty-second embodiment and further comprising the loss function is configured to only equal zero when a predicted output value of the neural network model exactly matches a labeled output value from the set of training samples.

[0443] A one hundred twenty-fourth embodiment, including the features of any of the one hundred sixth to one hundred twenty-third embodiments and further comprising the loss function is selected from the group consisting of a quadratic loss function and a cross-entropy loss function.

[0444] A one hundred twenty-fifth embodiment, including the features of any of the one hundred sixth to one hundred twenty-fourth embodiments and further comprising, at the current time iteration, obtaining the updated set of weights is based, at least in part, on a learning rate at the current time iteration.

[0445] A one hundred twenty-sixth embodiment, including the features of the one hundred twenty-fifth embodiment and further comprising the learning rate is timeinvariant.

[0446] A one hundred twenty-seventh embodiment, including the features of any of the one hundred sixth to one hundred twenty-sixth embodiments and further comprising updating the gradient of the potential function at the current time iteration is performed using at least one system selected from the group consisting of Adam, AdamW, AdaGrad, RMSProp, Momentum, and WeightDecay.

[0447] A one hundred twenty-eighth embodiment, including the features of any of the one hundred sixth to one hundred twenty-sixth embodiments and further comprising the gradient of the potential function at the current time iteration is obtained recursively based on the gradient of the potential function at a previous time iteration.

[0448] A one hundred twenty-ninth embodiment, including the features of the one hundred twenty-eighth embodiment and further comprising training the neural network model at the current time iteration further comprises storing the gradient of the potential function at the current time iteration.

[0449] A one hundred thirtieth embodiment, including the features of any of the one hundred sixth to one hundred twenty-ninth embodiments and further comprising, after the plurality of time iterations, the potential function is minimized via implicit regularization.

[0450] A one hundred thirty-first embodiment, including the features of any of the one hundred sixth to one hundred thirtieth embodiments and further comprising the potential function is a strictly convex function.

[0451] A one hundred thirty-second embodiment, including the features of the one hundred thirty-first embodiment and further comprising the potential function is a norm function.

[0452] A one hundred thirty-third embodiment, including the features of the one hundred thirty-first or one hundred thirty-second embodiment and further comprising the potential function is selected from the group consisting of: an £qnorm function where q » 1; and an £1+snorm function where 1 » E > 0.

[0453] A one hundred thirty-fourth embodiment, including the features of any of the one hundred sixth to one hundred thirty-third embodiments and further comprising training the neural network model comprises at least one selected from the group consisting of distributed training, batch training, synchronous training, ensemble learning, meta learning, transfer learning, and continual learning.

[0454] A one hundred thirty-fifth embodiment, including the features of any of the one hundred sixth to one hundred thirty-fourth embodiments and further comprising the inverse mapping of the gradient of the potential function is a many-to-one function.

[0455] A one hundred thirty-sixth embodiment, including the features of any of the one hundred sixth to one hundred thirty-fifth embodiments and further comprising a data size of each individual weight of the current set of weights is quantized to a single bit

[0456] A one hundred thirty-seventh embodiment, including the features of any of the one hundred sixth to one hundred thirty-sixth embodiments and further comprising the potential function is selected to enable the bimodality of the distribution of the trained set of weights.

[0457] A one hundred thirty-eighth embodiment, including the features of the one hundred thirty-seventh embodiment and further comprising the potential function is a rigid cage.

[0458] A one hundred thirty-ninth embodiment comprising a computing system for training-concurrent pruning. The computing system includes a network interface; a memory storing instructions; and one or more processors configured to communicate with the network interface and the memory, the one or more processors further configured to execute the instructions to perform operations. The operations include obtaining a pretrained neural network model, wherein the pretrained neural network model comprises initial values for a potential function and a set of weights. The operations include training a neural network model based upon the pretrained neural network model and a set of training samples, wherein training the neural network model comprises, for each of a plurality of time iterations: calculating loss for a current neural network model having a current set of weights at a current time iteration using a loss function; updating a gradient of the potential function at the current time iteration according to a gradient of the loss; updating an inverse mapping of the gradient of the potential function; creating an updated neural network model including an updated set of weights for a next time iteration according to the gradient of the potential function and the inverse mapping of the gradient of the potential function, wherein the updated set of weights is obtained by pruning at least a portion of the current set of weights of the current neural network model; and storing the updated set of weights.

[0459] A one hundred fortieth embodiment, including the features of the one hundred thirty-ninth embodiment and further comprising the trained neural network model is a multi-layered neural network model.

[0460] A one hundred forty-first embodiment, including the features of the one hundred fortieth embodiment and further comprising the multi-layered neural network model is selected from the group consisting of a convolutional neural network model, a recursive neural network model, a multi-layer perceptron, and a generative adversarial network.

[0461] A one hundred forty-second embodiment, including the features of the one hundred fortieth embodiment and further comprising the multi-layered neural network model is a large language model.

[0462] A one hundred forty-third embodiment, including the features of any of the one hundred thirty-ninth to one hundred forty-second embodiments and further comprising the updated set of weights is further obtained based on at least one additional vector comprising a current set of auxiliary variables for the current neural network model.

[0463] A one hundred forty-fourth embodiment, including the features of the one hundred forty-third embodiment and further comprising each sample of the set of training samples corresponds to a different auxiliary variable of the current set of auxiliary variables.

[0464] A one hundred forty-fifth embodiment, including the features of the one hundred forty-third or one hundred forty-fourth embodiment and further comprising obtaining the updated set of weights for the next time iteration comprises obtaining an updated set of auxiliary variables based on the current set of weights for the current time iteration.

[0465] A one hundred forty-sixth embodiment, including the features of the one hundred forty-fifth embodiment and further comprising each auxiliary variable of the current set of auxiliary variables is obtained based on a regularization parameter and the loss function evaluated for the current set of weights at the current time iteration.

[0466] A one hundred forty-seventh embodiment, including the features of the one hundred forty-fifth or one hundred forty-sixth embodiment and further comprising aregularizer mirror descent (RMD) process is used to obtain at least one selected from the group consisting of the updated set of weights; and the current set of auxiliary variables.

[0467] A one hundred forty-eighth embodiment, including the features of any of the one hundred forty-fifth to one hundred forty-seventh embodiments and further comprising iterative updates are used to obtain at least one selected from the group consisting of the updated set of weights; and the updated set of auxiliary variables.

[0468] A one hundred forty-ninth embodiment, including the features of any of the one hundred forty-third to one hundred forty-eighth embodiments and further comprising each of the current set of weights and the current set of auxiliary variables is initialized to at least one selected from the group consisting of: a zero vector and a non-zero vector comprising random numbers within a predetermined threshold of zero.

[0469] A one hundred fiftieth embodiment, including the features of any of the one hundred forty-third to one hundred forty-ninth embodiments and further comprising the loss function comprises a constraint-enforcing loss computed from the current set of auxiliary variables.

[0470] A one hundred fifty-first embodiment, including the features of the one hundred fiftieth embodiment and further comprising the constraint-enforcing loss is applied to prevent overfitting the neural network model to the set of training samples.

[0471] A one hundred fifty-second embodiment, including the features of any of the one hundred thirty-ninth to one hundred forty-second embodiments and further comprising a stochastic mirror descent (SMD) process is used to obtain the updated set of weights.

[0472] A one hundred fifty-third embodiment, including the features of any of the one hundred thirty-ninth to one hundred fifty-second embodiments and further comprising the loss function includes a Kurtosis term.

[0473] A one hundred fifty-fourth embodiment, including the features of the one hundred fifty-third embodiment and further comprising the Kurtosis term is a function of the current set of weights.

[0474] A one hundred fifty-fifth embodiment, including the features of any of the one hundred thirty-ninth to one hundred fifty-fourth embodiments and further comprising supervised learning is used to train the neural network model.

[0475] A one hundred fifty-sixth embodiment, including the features of the one hundred fifty-fifth embodiment and further comprising the loss function is configured to only equal zero when a predicted output value of the neural network model exactly matches a labeled output value from the set of training samples.

[0476] A one hundred fifty-seventh embodiment, including the features of any of the one hundred thirty-ninth to one hundred fifty-sixth embodiments and further comprising the loss function is selected from the group consisting of a quadratic loss function and a cross-entropy loss function.

[0477] A one hundred fifty-eighth embodiment, including the features of any of the one hundred thirty-ninth to one hundred fifty-seventh embodiments and further comprising, at the current time iteration, obtaining the updated set of weights is based, at least in part, on a learning rate at the current time iteration.

[0478] A one hundred fifty-ninth embodiment, including the features of the one hundred fifty-eighth embodiment and further comprising the learning rate is time-invariant.

[0479] A one hundred sixtieth embodiment, including the features of any of the one hundred thirty-ninth to one hundred fifty-ninth embodiments and further comprising updating the gradient of the potential function at the current time iteration is performed using at least one system selected from the group consisting of Adam, AdamW, AdaGrad, RMSProp, Momentum, and WeightDecay.

[0480] A one hundred sixty-first embodiment, including the features of any of the one hundred thirty-ninth to one hundred fifty-ninth embodiments and further comprising the gradient of the potential function at the current time iteration is obtained recursively based on the gradient of the potential function at a previous time iteration.

[0481] A one hundred sixty-second embodiment, including the features of the one hundred sixty-first embodiment and further comprising training the neural network model at the current time iteration further comprises storing the gradient of the potential function at the current time iteration.

[0482] A one hundred sixty-third embodiment, including the features of any of the one hundred thirty-ninth to one hundred sixty-second embodiments and further comprising, after the plurality of time iterations, the potential function is minimized via implicit regularization.

[0483] A one hundred sixty-fourth embodiment, including the features of any of the one hundred thirty-ninth to one hundred sixty-third embodiments and further comprising the potential function is a strictly convex function.

[0484] A one hundred sixty-fifth embodiment, including the features of the one hundred sixty-fourth embodiment and further comprising the potential function is a norm function.

[0485] A one hundred sixty-sixth embodiment, including the features of the one hundred sixty-fourth or one hundred sixty-fifth embodiment and further comprising the potential function is selected from the group consisting of: an £qnorm function where q » 1; and an £1+enorm function where 1 » s > 0.

[0486] A one hundred sixty-seventh embodiment, including the features of any of the one hundred thirty-ninth to one hundred sixty-sixth embodiments and further comprising training the neural network model comprises at least one selected from the group consisting of distributed training, batch training, synchronous training, ensemble learning, meta learning, transfer learning, and continual learning.

[0487] A one hundred sixty-eighth embodiment, including the features of any of the one hundred thirty-ninth to one hundred sixty-seventh embodiments and further comprising the inverse mapping of the gradient of the potential function is a many-to-one function.

[0488] A one hundred sixty-ninth embodiment, including the features of any of the one hundred thirty-ninth to one hundred sixty-eighth embodiments and further comprising obtaining the updated set of weights bimodally concentrates a distribution of the set of weights.

[0489] A one hundred seventieth embodiment, including the features of the one hundred sixty-ninth embodiment and further comprising quantizing the neural network model after the plurality of time iterations is performed according to a bimodality of the distribution of the set of weights.

[0490] A one hundred seventy-first embodiment, including the features of the one hundred sixty-ninth or one hundred seventieth embodiment and further comprising a data size of each individual weight of the current set of weights is quantized to a single bit.

[0491] A one hundred seventy-second embodiment, including the features of any of the one hundred thirty-ninth to one hundred seventy-first embodiments and further comprising the potential function is selected to induce sparsity within the set of weights.

[0492] A one hundred seventy-third embodiment, including the features of the one hundred seventy-second embodiment and further comprising the potential function is selected to induce group sparsity within a trained set of weights for the trained neural network model.

[0493] A one hundred seventy-fourth embodiment, including the features of the one hundred seventy-second or one hundred seventy-third embodiment and further comprising the potential function is an elastic net.

[0494] A one hundred seventy-fifth embodiment comprising a computing system for training-concurrent quantization. The computing system includes a network interface; a memory storing instructions; and one or more processors configured to communicate with the network interface and the memory, the one or more processors further configured to execute the instructions to perform operations. The operations include obtaining a pretrained neural network model, wherein the pretrained neural network model comprises initial values for a potential function and a set of weights. The operations include training a neural network model based upon the pretrained neural network model and a set of training samples, wherein training the neural network model comprises, for each of a plurality of time iterations: calculating loss for a current neural network model having a current set of weights at a current time iteration using a loss function; updating a gradient of the potential function at the current time iteration according to a gradient of the loss; updating an inverse mapping of the gradient of the potential function; creating an updated neural network model including an updated set of weights for a next time iteration according to the gradient of the potential function and the inverse mapping of the gradient of the potential function, wherein creating the updated neural network model bimodally concentrates a distribution of the updated set of weights; and storing the updated set of weights. The operations include quantizing the trained neural network model after the plurality of time iterations, wherein quantizing the trained neural network model is performed according to a bimodality of the distribution of a trained set of weights.

[0495] A one hundred seventy-sixth embodiment, including the features of the one hundred seventy-fifth embodiment and further comprising the trained neural network model is a multi-layered neural network model.

[0496] A one hundred seventy-seventh embodiment, including the features of the one hundred seventy-sixth embodiment and further comprising the multi-layered neural network model is selected from the group consisting of a convolutional neural network model, a recursive neural network model, a multi-layer perceptron, and a generative adversarial network.

[0497] A one hundred seventy-eighth embodiment, including the features of the one hundred seventy-sixth embodiment and further comprising the multi-layered neural network model is a large language model.

[0498] A one hundred seventy-ninth embodiment, including the features of any of the one hundred seventy-fifth to one hundred seventy-eighth embodiments and further comprising the updated set of weights is further obtained based on at least one additional vector comprising a current set of auxiliary variables for the current neural network model.

[0499] A one hundred eightieth embodiment, including the features of the one hundred seventy-ninth embodiment and further comprising each sample of the set of training samples corresponds to a different auxiliary variable of the current set of auxiliary variables.

[0500] A one hundred eighty-first embodiment, including the features of the one hundred seventy-ninth or one hundred eightieth embodiment and further comprising obtaining the updated set of weights for the next time iteration comprises obtaining an updated set of auxiliary variables based on the current set of weights for the current time iteration.

[0501] A one hundred eighty-second embodiment, including the features of the one hundred eighty-first embodiment and further comprising each auxiliary variable of the current set of auxiliary variables is obtained based on a regularization parameter and the loss function evaluated for the current set of weights at the current time iteration.

[0502] A one hundred eighty-third embodiment, including the features of the one hundred eighty-first or one hundred eighty-second embodiment and further comprising aregularizer mirror descent (RMD) process is used to obtain at least one selected from the group consisting of the updated set of weights; and the current set of auxiliary variables.

[0503] A one hundred eighty-fourth embodiment, including the features of any of the one hundred eighty-first to one hundred eighty-third embodiments and further comprising iterative updates are used to obtain at least one selected from the group consisting of the updated set of weights; and the updated set of auxiliary variables.

[0504] A one hundred eighty-fifth embodiment, including the features of any of the one hundred seventy-ninth to one hundred eighty-fourth embodiments and further comprising each of the current set of weights and the current set of auxiliary variables is initialized to at least one selected from the group consisting of: a zero vector and a nonzero vector comprising random numbers within a predetermined threshold of zero.

[0505] A one hundred eighty-sixth embodiment, including the features of any of the one hundred seventy-ninth to one hundred eighty-fifth embodiments and further comprising the loss function comprises a constraint-enforcing loss computed from the current set of auxiliary variables.

[0506] A one hundred eighty-seventh embodiment, including the features of the one hundred eighty-sixth embodiment and further comprising the constraint-enforcing loss is applied to prevent overfitting the neural network model to the set of training samples.

[0507] A one hundred eighty-eighth embodiment, including the features of any of the one hundred seventy-fifth to one hundred seventy-eighth embodiments and further comprising a stochastic mirror descent (SMD) process is used to obtain the updated set of weights.

[0508] A one hundred eighty-ninth embodiment, including the features of any of the one hundred seventy-fifth to one hundred eighty-eighth embodiments and further comprising the loss function includes a Kurtosis term.

[0509] A one hundred ninetieth embodiment, including the features of the one hundred eighty-ninth embodiment and further comprising the Kurtosis term is a function of the current set of weights.

[0510] A one hundred ninety-first embodiment, including the features of any of the one hundred seventy-fifth to one hundred ninetieth embodiments and further comprising supervised learning is used to train the neural network model.

[0511] A one hundred ninety-second embodiment, including the features of the one hundred ninety-first embodiment and further comprising the loss function is configured to only equal zero when a predicted output value of the neural network model exactly matches a labeled output value from the set of training samples.

[0512] A one hundred ninety-third embodiment, including the features of any of the one hundred seventy-fifth to one hundred ninety-second embodiments and further comprising the loss function is selected from the group consisting of a quadratic loss function and a cross-entropy loss function.

[0513] A one hundred ninety-fourth embodiment, including the features of any of the one hundred seventy-fifth to one hundred ninety-third embodiments and further comprising, at the current time iteration, obtaining the updated set of weights is based, at least in part, on a learning rate at the current time iteration.

[0514] A one hundred ninety-fifth embodiment, including the features of the one hundred ninety-fourth embodiment and further comprising the learning rate is timeinvariant.

[0515] A one hundred ninety-sixth embodiment, including the features of any of the one hundred seventy-fifth to one hundred ninety-fifth embodiments and further comprising updating the gradient of the potential function at the current time iteration is performed using at least one system selected from the group consisting of Adam, AdamW, AdaGrad, RMSProp, Momentum, and WeightDecay.

[0516] A one hundred ninety-seventh embodiment, including the features of any of the one hundred seventy-fifth to one hundred ninety-fifth embodiments and further comprising the gradient of the potential function at the current time iteration is obtained recursively based on the gradient of the potential function at a previous time iteration.

[0517] A one hundred ninety-eighth embodiment, including the features of the one hundred ninety-seventh embodiment and further comprising training the neural network model at the current time iteration further comprises storing the gradient of the potential function at the current time iteration.

[0518] A one hundred ninety-ninth embodiment, including the features of any of the one hundred seventy-fifth to one hundred ninety-eighth embodiments and further comprising, after the plurality of time iterations, the potential function is minimized via implicit regularization.

[0519] A two hundredth embodiment, including the features of any of the one hundred seventy-fifth to one hundred ninety-ninth embodiments and further comprising the potential function is a strictly convex function.

[0520] A two hundred first embodiment, including the features of the two hundredth embodiment and further comprising the potential function is a norm function.

[0521] A two hundred second embodiment, including the features of the two hundredth or two hundred first embodiment and further comprising the potential function is selected from the group consisting of: an £qnorm function where q » 1; and an £1+enorm function where 1 » E > 0.

[0522] A two hundred third embodiment, including the features of any of the one hundred seventy-fifth to two hundred second embodiments and further comprising training the neural network model comprises at least one selected from the group consisting of distributed training, batch training, synchronous training, ensemble learning, meta learning, transfer learning, and continual learning.

[0523] A two hundred fourth embodiment, including the features of any of the one hundred seventy-fifth to two hundred third embodiments and further comprising the inverse mapping of the gradient of the potential function is a many-to-one function.

[0524] A two hundred fifth embodiment, including the features of any of the one hundred seventy-fifth to two hundred fourth embodiments and further comprising a data size of each individual weight of the current set of weights is quantized to a single bit.

[0525] A two hundred sixth embodiment, including the features of any of the one hundred seventy-fifth to two hundred fifth embodiments and further comprising the potential function is selected to enable the bimodality of the distribution of the trained set of weights.

[0526] A two hundred seventh embodiment, including the features of the two hundred sixth embodiment and further comprising the potential function is a rigid cage.

[0527] While specific methods of training models are discussed above, many different methods of training models can be implemented in accordance with many different embodiments of the invention. It is therefore to be understood that the present invention may be practiced in ways other than specifically described, without departing from the scope and spirit of the present invention. Thus, embodiments of the present invention should be considered in all respects as illustrative and not restrictive. Accordingly, the scope of the invention should be determined not by the embodiments illustrated, but by the appended claims and their equivalents.

Claims

WHAT IS CLAIMED IS:

1. A method for training-concurrent pruning, performed by one or more processors, comprising:obtaining a pretrained neural network model, wherein the pretrained neural network model comprises initial values for a potential function and a set of weights; and training a neural network model based upon the pretrained neural network model and a set of training samples, wherein training the neural network model comprises, for each of a plurality of time iterations:calculating loss for a current neural network model having a current set of weights at a current time iteration using a loss function;updating a gradient of the potential function at the current time iteration according to a gradient of the loss;updating an inverse mapping of the gradient of the potential function; creating an updated neural network model including an updated set of weights for a next time iteration according to the gradient of the potential function and the inverse mapping of the gradient of the potential function, wherein the updated set of weights is obtained by pruning at least a portion of the current set of weights of the current neural network model; andstoring the updated set of weights.

2. The method of claim 1, wherein the trained neural network model is a multi-layered neural network model.

3. The method of claim 2, wherein the multi-layered neural network model is selected from the group consisting of a convolutional neural network model, a recursive neural network model, a multi-layer perceptron, and a generative adversarial network.

4. The method of claim 2, wherein the multi-layered neural network model is a large language model.

5. The method of any of claims 1-4, wherein the updated set of weights is further obtained based on at least one additional vector comprising a current set of auxiliary variables for the current neural network model.

6. The method of claim 5, wherein each sample of the set of training samples corresponds to a different auxiliary variable of the current set of auxiliary variables.

7. The method of claim 5 or claim 6, wherein obtaining the updated set of weights for the next time iteration comprises obtained an updated set of auxiliary variables based on the current set of weights for the current time iteration.

8. The method of claim 7, wherein each auxiliary variable of the current set of auxiliary variables is obtained based on a regularization parameter and the loss function evaluated for the current set of weights at the current time iteration.

9. The method of claim 7 or claim 8, wherein a regularizer mirror descent (RMD) process is used to obtain at least one selected from the group consisting of the updated set of weights; and the current set of auxiliary variables.

10. The method of any of claims 7-9, wherein iterative updates are used to obtain at least one selected from the group consisting of the updated set of weights; and the updated set of auxiliary variables.

11. The method of any of claims 5-10, wherein each of the current set of weights and the current set of auxiliary variables is initialized to at least one selected from the group consisting of: a zero vector and a non-zero vector comprising random numbers within a predetermined threshold of zero.

12. The method of any of claims 5-11, wherein the loss function comprises a constraint-enforcing loss computed from the current set of auxiliary variables.

13. The method of claim 12, wherein the constraint-enforcing loss is applied to prevent overfitting the neural network model to the set of training samples.

14. The method of any of claims 1-4, wherein a stochastic mirror descent (SMD) process is used to obtain the updated set of weights.

15. The method of any of claims 1-14, wherein the loss function includes a Kurtosis term.

16. The method of claim 15, wherein the Kurtosis term is a function of the current set of weights.

17. The method of any of claims 1 -15, wherein supervised learning is used to train the neural network model.

18. The method of claim 17, wherein the loss function is configured to only equal when a predicted output value of the neural network model exactly matches a labeled output value from the set of training samples.

19. The method of any of claims 1-18, wherein the loss function is selected from the group consisting of a quadratic loss function and a cross-entropy loss function.

20. The method of any of claims 1-19, wherein, at the current time iteration, obtaining the updated set of weights is based, at least in part, on a learning rate at the current time iteration.

21. The method of claim 20, wherein the learning rate is time-invariant.

22. The method of any of claims 1-21, wherein updating the gradient of the potential function at the current time iteration is performed using at least one system selected from the group consisting of Adam, AdamW, AdaGrad, RMSProp, Momentum, and WeightDecay.

23. The method of any of claims 1 -21, wherein the gradient of the potential function at the current time iteration is obtained recursively based on the gradient of the potential function at a previous time iteration.

24. The method of claim 23, wherein training the neural network model at the current time iteration further comprises storing the gradient of the potential function at the current time iteration.

25. The method of any of claims 1 -24, wherein, after the plurality of time iterations, the potential function is minimized via implicit regularization.

26. The method of any of claims 1 -25, wherein the potential function is a strictly convex function.

27. The method of claim 26, wherein the potential function is a norm function.

28. The method of claim 26 or claim 27, wherein the potential function is selected from the group consisting of:an £qnorm function where q »1; andan £1+enorm function where 1 » e > 0.

29. The method of any of claims 1-28, wherein training the neural network model comprises at least one selected from the group consisting of distributed training, batch training, synchronous training, ensemble learning, meta learning, transfer learning, and continual learning.

30. The method of any of claims 1-29, wherein the inverse mapping of the gradient of the potential function is a many-to-one function.

31. The method of any of claims 1-30, wherein obtaining the updated set of weights bimodally concentrates a distribution of the set of weights.

32. The method of claim 31, wherein quantizing the neural network model after the plurality of time iterations is performed according to a bimodality of the distribution of the set of weights.

33. The method of claim 31 or claim 32, wherein a data size of each individual weight of the current set of weights is quantized to a single bit.

34. The method of any of claims 1-33, wherein the potential function is selected to induce sparsity within the set of weights.

35. The method of claim 34, wherein the potential function is selected to induce group sparsity within a trained set of weights for the trained neural network model.

36. The method of claim 34 or claim 35, wherein the potential function is an elastic net.

37. A method for training-concurrent quantization, performed by one or more processors, comprising:obtaining a pretrained neural network model, wherein the pretrained neural network model comprises initial values for a potential function and a set of weights; training a neural network model based upon the pretrained neural network model and a set of training samples, wherein training the neural network model comprises, for each of a plurality of time iterations:calculating loss for a current neural network model having a current set of weights at a current time iteration using a loss function;updating a gradient of the potential function at the current time iteration according to a gradient of the loss;updating an inverse mapping of the gradient of the potential function; creating an updated neural network model including an updated set of weights for a next time iteration according to the gradient of the potential function and the inverse mapping of the gradient of the potential function, wherein creating the updated neural network model bimodally concentrates a distribution of the updated set of weights; andstoring the updated set of weights; andquantizing the trained neural network model after the plurality of time iterations, wherein quantizing the trained neural network model is performed according to a bimodality of the distribution of a trained set of weights.

38. The method of claim 37, wherein the trained neural network model is a multilayered neural network model.

39. The method of claim 38, wherein the multi-layered neural network model is selected from the group consisting of a convolutional neural network model, a recursive neural network model, a multi-layer perceptron, and a generative adversarial network.

40. The method of claim 38, wherein the multi-layered neural network model is a large language model.

41. The method of any of claims 37-40, wherein the updated set of weights is further obtained based on at least one additional vector comprising a current set of auxiliary variables for the current neural network model.

42. The method of claim 41, wherein each sample of the set of training samples corresponds to a different auxiliary variable of the current set of auxiliary variables.

43. The method of claim 41 or claim 42, wherein obtaining the updated set of weights for the next time iteration comprises obtaining an updated set of auxiliary variables based on the current set of weights for the current time iteration.

44. The method of claim 43, wherein each auxiliary variable of the current set of auxiliary variables is obtained based on a regularization parameter and the loss function evaluated for the current set of weights at the current time iteration.

45. The method of claim 43 or claim 44, wherein a regularizer mirror descent (RMD) process is used to obtain at least one selected from the group consisting of the updated set of weights; and the current set of auxiliary variables.

46. The method of any of claims 43-45, wherein iterative updates are used to obtain at least one selected from the group consisting of the updated set of weights; and the updated set of auxiliary variables.

47. The method of any of claims 41 -46, wherein each of the current set of weights and the current set of auxiliary variables is initialized to at least one selected from the group consisting of: a zero vector and a non-zero vector comprising random numbers within a predetermined threshold of zero.

48. The method of any of claims 41-47, wherein the loss function comprises a constraint-enforcing loss computed from the current set of auxiliary variables.

49. The method of claim 48, wherein the constraint-enforcing loss is applied to prevent overfitting the neural network model to the set of training samples.

50. The method of any of claims 37-40, wherein a stochastic mirror descent (SMD) process is used to obtain the updated set of weights.

51. The method of any of claims 37-50, wherein the loss function includes a Kurtosis term.

52. The method of claim 51, wherein the Kurtosis term is a function of the current set of weights.

53. The method of any of claims 37-51, wherein supervised learning is used to train the neural network model.

54. The method of claim 53, wherein the loss function is configured to only equal zero when a predicted output value of the neural network model exactly matches a labeled output value from the set of training samples.

55. The method of any of claims 37-54, wherein the loss function is selected from the group consisting of a quadratic loss function and a cross-entropy loss function.

56. The method of any of claims 37-55, wherein, at the current time iteration, obtaining the updated set of weights is based, at least in part, on a learning rate at the current time iteration.

57. The method of claim 56, wherein the learning rate is time-invariant.

58. The method of any of claims 37-57, wherein updating the gradient of the potential function at the current time iteration is performed using at least one system selected from the group consisting of Adam, AdamW, AdaGrad, RMSProp, Momentum, and WeightDecay.

59. The method of any of claims 37-57, wherein the gradient of the potential function at the current time iteration is obtained recursively based on the gradient of the potential function at a previous time iteration.

60. The method of claim 59, wherein training the neural network model at the current time iteration further comprises storing the gradient of the potential function at the current time iteration.

61. The method of any of claims 37-60, wherein, after the plurality of time iterations, the potential function is minimized via implicit regularization.

62. The method of any of claims 37-61, wherein the potential function is a strictly convex function.

63. The method of claim 62, wherein the potential function is a norm function.

64. The method of claim 62 or claim 63, wherein the potential function is selected from the group consisting of:an £qnorm function where q »1; andan £1+enorm function where 1 » e > 0.

65. The method of any of claims 37-64, wherein training the neural network model comprises at least one selected from the group consisting of distributed training, batch training, synchronous training, ensemble learning, meta learning, transfer learning, and continual learning.

66. The method of any of claims 37-65, wherein the inverse mapping of the gradient of the potential function is a many-to-one function.

67. The method of any of claims 37-66, wherein a data size of each individual weight of the current set of weights is quantized to a single bit.

68. The method of any of claims 37-67, wherein the potential function is selected to enable the bimodality of the distribution of the trained set of weights.

69. The method of claim 68, wherein the potential function is a rigid cage.

70. A non-transitory machine-readable medium storing instructions that, when executed by a processor, are configured to cause the processor to perform operations for training-concurrent pruning, the operations comprising:obtaining a pretrained neural network model, wherein the pretrained neural network model comprises initial values for a potential function and a set of weights; and training a neural network model based upon the pretrained neural network model and a set of training samples, wherein training the neural network model comprises, for each of a plurality of time iterations:calculating loss for a current neural network model having a current set of weights at a current time iteration using a loss function;updating a gradient of the potential function at the current time iteration according to a gradient of the loss;updating an inverse mapping of the gradient of the potential function; creating an updated neural network model including an updated set of weights for a next time iteration according to the gradient of the potential function and the inverse mapping of the gradient of the potential function, wherein the updated set of weights is obtained by pruning at least a portion of the current set of weights of the current neural network model; andstoring the updated set of weights.

71. The non-transitory machine-readable medium of claim 70, wherein the trained neural network model is a multi-layered neural network model.

72. The non-transitory machine-readable medium of claim 71, wherein the multilayered neural network model is selected from the group consisting of a convolutional neural network model, a recursive neural network model, a multi-layer perceptron, and a generative adversarial network.

73. The non-transitory machine-readable medium of claim 71, wherein the multilayered neural network model is a large language model.

74. The non-transitory machine-readable medium of any of claims 70-73, wherein the updated set of weights is further obtained based on at least one additional vector comprising a current set of auxiliary variables for the current neural network model.

75. The non-transitory machine-readable medium of claim 74, wherein each sample of the set of training samples corresponds to a different auxiliary variable of the current set of auxiliary variables.

76. The non-transitory machine-readable medium of claim 74 or claim 75, wherein obtaining the updated set of weights for the next time iteration comprises obtaining an updated set of auxiliary variables based on the current set of weights for the current time iteration.

77. The non-transitory machine-readable medium of claim 76, wherein each auxiliary variable of the current set of auxiliary variables is obtained based on a regularization parameter and the loss function evaluated for the current set of weights at the current time iteration.

78. The non-transitory machine-readable medium of claim 76 or claim 77, wherein a regularizer mirror descent (RMD) process is used to obtain at least one selected from the group consisting of the updated set of weights; and the current set of auxiliary variables.

79. The non-transitory machine-readable medium of any of claims 76-78, wherein iterative updates are used to obtain at least one selected from the group consisting of the updated set of weights; and the updated set of auxiliary variables.

80. The non-transitory machine-readable medium of any of claims 74-79, wherein each of the current set of weights and the current set of auxiliary variables is initialized to at least one selected from the group consisting of: a zero vector and a non-zero vector comprising random numbers within a predetermined threshold of zero.

81. The non-transitory machine-readable medium of any of claims 74-80, wherein the loss function comprises a constraint-enforcing loss computed from the current set of auxiliary variables.

82. The non-transitory machine-readable medium of claim 81, wherein the constraint-enforcing loss is applied to prevent overfitting the neural network model to the set of training samples.

83. The non-transitory machine-readable medium of any of claims 70-73, wherein a stochastic mirror descent (SMD) process is used to obtain the updated set of weights.

84. The non-transitory machine-readable medium of any of claims 70-83, wherein the loss function includes a Kurtosis term.

85. The non-transitory machine-readable medium of claim 84, wherein the Kurtosis term is a function of the current set of weights.

86. The non-transitory machine-readable medium of any of claims 70-85, wherein supervised learning is used to train the neural network model.

87. The non-transitory machine-readable medium of claim 86, wherein the loss function is configured to only equal zero when a predicted output value of the neural network model exactly matches a labeled output value from the set of training samples.

88. The non-transitory machine-readable medium of any of claims 70-87, wherein the loss function is selected from the group consisting of a quadratic loss function and a cross-entropy loss function.

89. The non-transitory machine-readable medium of any of claims 70-88, wherein, at the current time iteration, obtaining the updated set of weights is based, at least in part, on a learning rate at the current time iteration.

90. The non-transitory machine-readable medium of claim 89, wherein the learning rate is time-invariant.

91. The non-transitory machine-readable medium of any of claims 70-90, wherein updating the gradient of the potential function at the current time iteration is performed using at least one system selected from the group consisting of Adam, AdamW, AdaGrad, RMSProp, Momentum, and WeightDecay.

92. The non-transitory machine-readable medium of any of claims 70-90, wherein the gradient of the potential function at the current time iteration is obtained recursively based on the gradient of the potential function at a previous time iteration.

93. The non-transitory machine-readable medium of claim 92, wherein training the neural network model at the current time iteration further comprises storing the gradient of the potential function at the current time iteration.

94. The non-transitory machine-readable medium of any of claims 70-93, wherein, after the plurality of time iterations, the potential function is minimized via implicit regularization.

95. The non-transitory machine-readable medium of any of claims 70-94, wherein the potential function is a strictly convex function.

96. The non-transitory machine-readable medium of claim 95, wherein the potential function is a norm function.

97. The non-transitory machine-readable medium of claim 95 or claim 96, wherein the potential function is selected from the group consisting of:an £qnorm function where q » 1; andan £1+enorm function where 1 » E > 0.

98. The non-transitory machine-readable medium of any of claims 70-97, wherein training the neural network model comprises at least one selected from the group consisting of distributed training, batch training, synchronous training, ensemble learning, meta learning, transfer learning, and continual learning.

99. The non-transitory machine-readable medium of any of claims 70-98, wherein the inverse mapping of the gradient of the potential function is a many-to-one function.

100. The non-transitory machine-readable medium of any of claims 70-99, wherein obtaining the updated set of weights bimodally concentrates a distribution of the set of weights.

101. The non-transitory machine-readable medium of claim 100, wherein quantizing the neural network model after the plurality of time iterations is performed according to a bimodality of the distribution of the set of weights.

102. The non-transitory machine-readable medium of claim 100 or claim 101, wherein a data size of each individual weight of the current set of weights is quantized to a single bit.

103. The non-transitory machine-readable medium of any of claims 70-102, wherein the potential function is selected to induce sparsity within the set of weights.

104. The non-transitory machine-readable medium of claim 103, wherein the potential function is selected to induce group sparsity within a trained set of weights for the trained neural network model.

105. The non-transitory machine-readable medium of claim 103 or claim 104, wherein the potential function is an elastic net.

106. A non-transitory machine-readable medium storing instructions that, when executed by a processor, are configured to cause the processor to perform operations for training-concurrent quantization, the operations comprising:obtaining a pretrained neural network model, wherein the pretrained neural network model comprises initial values for a potential function and a set of weights; training a neural network model based upon the pretrained neural network model and a set of training samples, wherein training the neural network model comprises, for each of a plurality of time iterations:calculating loss for a current neural network model having a current set of weights at a current time iteration using a loss function;updating a gradient of the potential function at the current time iteration according to a gradient of the loss;updating an inverse mapping of the gradient of the potential function; creating an updated neural network model including an updated set of weights for a next time iteration according to the gradient of the potential function and the inverse mapping of the gradient of the potential function, wherein creating the updated neural network model bimodally concentrates a distribution of the updated set of weights; andstoring the updated set of weights; andquantizing the trained neural network model after the plurality of time iterations, wherein quantizing the trained neural network model is performed according to a bimodality of the distribution of a trained set of weights.

107. The non-transitory machine-readable medium of claim 106, wherein the trained neural network model is a multi-layered neural network model.

108. The non-transitory machine-readable medium of claim 107, wherein the multilayered neural network model is selected from the group consisting of a convolutional neural network model, a recursive neural network model, a multi-layer perceptron, and a generative adversarial network.

109. The non-transitory machine-readable medium of claim 107, wherein the multilayered neural network model is a large language model.

110. The non-transitory machine-readable medium of any of claims 106-109, wherein the updated set of weights is further obtained based on at least one additional vector comprising a current set of auxiliary variables for the current neural network model.

111. The non-transitory machine-readable medium of claim 110, wherein each sample of the set of training samples corresponds to a different auxiliary variable of the current set of auxiliary variables.

112. The non-transitory machine-readable medium of claim 110 or claim 111, wherein obtaining the updated set of weights for the next time iteration comprises obtaining an updated set of auxiliary variables based on the current set of weights for the current time iteration.

113. The non-transitory machine-readable medium of claim 112, wherein each auxiliary variable of the current set of auxiliary variables is obtained based on a regularization parameter and the loss function evaluated for the current set of weights at the current time iteration.

114. The non-transitory machine-readable medium of claim 112 or claim 113, wherein a regularizer mirror descent (RMD) process is used to obtain at least one selected from the group consisting of the updated set of weights; and the current set of auxiliary variables.

115. The non-transitory machine-readable medium of any of claims 112-114, wherein iterative updates are used to obtain at least one selected from the group consisting of the updated set of weights; and the updated set of auxiliary variables.

116. The non-transitory machine-readable medium of any of claims 110-115, wherein each of the current set of weights and the current set of auxiliary variables is initialized to at least one selected from the group consisting of: a zero vector and a non-zero vector comprising random numbers within a predetermined threshold of zero.

117. The non-transitory machine-readable medium of any of claims 110-116, wherein the loss function comprises a constraint-enforcing loss computed from the current set of auxiliary variables.

118. The non-transitory machine-readable medium of claim 117, wherein the constraint-enforcing loss is applied to prevent overfitting the neural network model to the set of training samples.

119. The non-transitory machine-readable medium of any of claims 106-109, wherein a stochastic mirror descent (SMD) process is used to obtain the updated set of weights.

120. The non-transitory machine-readable medium of any of claims 106-119, wherein the loss function includes a Kurtosis term.

121. The non-transitory machine-readable medium of claim 120, wherein the Kurtosis term is a function of the current set of weights.

122. The non-transitory machine-readable medium of any of claims 106-121, wherein supervised learning is used to train the neural network model.

123. The non-transitory machine-readable medium of claim 122, wherein the loss function is configured to only equal zero when a predicted output value of the neural network model exactly matches a labeled output value from the set of training samples.

124. The non-transitory machine-readable medium of any of claims 106-123, wherein the loss function is selected from the group consisting of a quadratic loss function and a cross-entropy loss function.

125. The non-transitory machine-readable medium of any of claims 106-124, wherein, at the current time iteration, obtaining the updated set of weights is based, at least in part, on a learning rate at the current time iteration.

126. The non-transitory machine-readable medium of claim 125, wherein the learning rate is time-invariant.

127. The non-transitory machine-readable medium of any of claims 106-126, wherein updating the gradient of the potential function at the current time iteration is performed using at least one system selected from the group consisting of Adam, AdamW, AdaGrad, RMSProp, Momentum, and WeightDecay.

128. The non-transitory machine-readable medium of any of claims 106-126, wherein the gradient of the potential function at the current time iteration is obtained recursively based on the gradient of the potential function at a previous time iteration.

129. The non-transitory machine-readable medium of claim 128, wherein training the neural network model at the current time iteration further comprises storing the gradient of the potential function at the current time iteration.

130. The non-transitory machine-readable medium of any of claims 106-129, wherein, after the plurality of time iterations, the potential function is minimized via implicit regularization.

131. The non-transitory machine-readable medium of any of claims 106-130, wherein the potential function is a strictly convex function.

132. The non-transitory machine-readable medium of claim 131, wherein the potential function is a norm function.

133. The non-transitory machine-readable medium of claim 131 or claim 132, wherein the potential function is selected from the group consisting of:an Lqnorm function where q » 1; andan £1+enorm function where 1 » s > 0.

134. The non-transitory machine-readable medium of any of claims 106-133, wherein training the neural network model comprises at least one selected from the group consisting of distributed training, batch training, synchronous training, ensemble learning, meta learning, transfer learning, and continual learning.

135. The non-transitory machine-readable medium of any of claims 106-134, wherein the inverse mapping of the gradient of the potential function is a many-to-one function.

136. The non-transitory machine-readable medium of any of claims 106-135, wherein a data size of each individual weight of the current set of weights is quantized to a single bit.

137. The non-transitory machine-readable medium of any of claims 106-136, wherein the potential function is selected to enable the bimodality of the distribution of the trained set of weights.

138. The non-transitory machine-readable medium of claim 137, wherein the potential function is a rigid cage.

139. A computing system for training-concurrent pruning, comprising:a network interface;a memory storing instructions; andone or more processors configured to communicate with the network interface and the memory, the one or more processors further configured to execute the instructions to perform operations comprising:obtaining, using the network interface, a pretrained neural network model, wherein the pretrained neural network model comprises initial values for a potential function and a set of weights; andtraining a neural network model based upon the pretrained neural network model and a set of training samples, wherein training the neural network model comprises, for each of a plurality of time iterations:calculating loss for a current neural network model having a current set of weights at a current time iteration using a loss function; updating a gradient of the potential function at the current time iteration according to a gradient of the loss;updating an inverse mapping of the gradient of the potential function;creating, using the network interface an updated neural network model including an updated set of weights for a next time iteration according to the gradient of the potential function and the inverse mapping of the gradient of the potential function, wherein the updated set of weights is obtained by pruning at least a portion of the current set of weights of the current neural network model; andstoring the updated set of weights.

140. The computing system of claim 139, wherein the trained neural network model is a multi-layered neural network model.

141. The computing system of claim 140, wherein the multi-layered neural network model is selected from the group consisting of a convolutional neural network model, a recursive neural network model, a multi-layer perceptron, and a generative adversarial network.

142. The computing system of claim 140, wherein the multi-layered neural network model is a large language model.

143. The computing system of any of claims 139-142, wherein the updated set of weights is further obtained based on at least one additional vector comprising a current set of auxiliary variables for the current neural network model.

144. The computing system of claim 143, wherein each sample of the set of training samples corresponds to a different auxiliary variable of the current set of auxiliary variables.

145. The computing system of claim 143 or claim 144, wherein obtaining the updated set of weights for the next time iteration comprises obtaining an updated set of auxiliary variables based on the current set of weights for the current time iteration.

146. The computing system of claim 145, wherein each auxiliary variable of the current set of auxiliary variables is obtained based on a regularization parameter and the loss function evaluated for the current set of weights at the current time iteration.

147. The computing system of claim 145 or claim 146, wherein a regularizer mirror descent (RMD) process is used to obtain at least one selected from the group consisting of the updated set of weights; and the current set of auxiliary variables.

148. The computing system of any of claims 145-147, wherein iterative updates are used to obtain at least one selected from the group consisting of the updated set of weights; and the updated set of auxiliary variables.

149. The computing system of any of claims 143-148, wherein each of the current set of weights and the current set of auxiliary variables is initialized to at least one selected from the group consisting of: a zero vector and a non-zero vector comprising random numbers within a predetermined threshold of zero.

150. The computing system of any of claims 143-149, wherein the loss function comprises a constraint-enforcing loss computed from the current set of auxiliary variables.

151. The computing system of claim 150, wherein the constraint-enforcing loss is applied to prevent overfitting the neural network model to the set of training samples.

152. The computing system of any of claims 139-142, wherein a stochastic mirror descent (SMD) process is used to obtain the updated set of weights.

153. The computing system of any of claims 139-152, wherein the loss function includes a Kurtosis term.

154. The computing system of claim 153, wherein the Kurtosis term is a function of the current set of weights.

155. The computing system of any of claims 139-154, wherein supervised learning is used to train the neural network model.

156. The computing system of claim 155, wherein the loss function is configured to only equal zero when a predicted output value of the neural network model exactly matches a labeled output value from the set of training samples.

157. The computing system of any of claims 139-156, wherein the loss function is selected from the group consisting of a quadratic loss function and a cross-entropy loss function.

158. The computing system of any of claims 139-157, wherein, at the current time iteration, obtaining the updated set of weights is based, at least in part, on a learning rate at the current time iteration.

159. The computing system of claim 158, wherein the learning rate is time-invariant.

160. The computing system of any of claims 139-159, wherein updating the gradient of the potential function at the current time iteration is performed using at least one system selected from the group consisting of Adam, AdamW, AdaGrad, RMSProp, Momentum, and WeightDecay.

161. The computing system of any of claims 139-159, wherein the gradient of the potential function at the current time iteration is obtained recursively based on the gradient of the potential function at a previous time iteration.

162. The computing system of claim 161, wherein training the neural network model at the current time iteration further comprises storing the gradient of the potential function at the current time iteration.

163. The computing system of any of claims 139-162, wherein, after the plurality of time iterations, the potential function is minimized via implicit regularization.

164. The computing system of any of claims 139-163, wherein the potential function is a strictly convex function.

165. The computing system of claim 164, wherein the potential function is a norm function.

166. The computing system of claim 164 or claim 165, wherein the potential function is selected from the group consisting of:an £qnorm function where q » 1; andan £1+enorm function where 1 » E > 0.

167. The computing system of any of claims 139-166, wherein training the neural network model comprises at least one selected from the group consisting of distributed training, batch training, synchronous training, ensemble learning, meta learning, transfer learning, and continual learning.

168. The computing system of any of claims 139-167, wherein the inverse mapping of the gradient of the potential function is a many-to-one function.

169. The computing system of any of claims 139-168, wherein obtaining the updated set of weights bimodally concentrates a distribution of the set of weights.

170. The computing system of claim 169, wherein quantizing the neural network model after the plurality of time iterations is performed according to a bimodality of the distribution of the set of weights.

171. The computing system of claim 169 or claim 170, wherein a data size of each individual weight of the current set of weights is quantized to a single bit.

172. The computing system of any of claims 139-171, wherein the potential function is selected to induce sparsity within the set of weights.

173. The computing system of claim 172, wherein the potential function is selected to induce group sparsity within a trained set of weights for the trained neural network model.

174. The computing system of claim 172 or claim 173, wherein the potential function is an elastic net.

175. A computing system for training-concurrent quantization, comprising:a network interface;a memory storing instructions; andone or more processors configured to communicate with the network interface and the memory, the one or more processors further configured to execute the instructions to perform operations comprising:obtaining, using the network interface, a pretrained neural network model, wherein the pretrained neural network model comprises initial values for a potential function and a set of weights;training a neural network model based upon the pretrained neural network model and a set of training samples, wherein training the neural network model comprises, for each of a plurality of time iterations:calculating loss for a current neural network model having a current set of weights at a current time iteration using a loss function; updating a gradient of the potential function at the current time iteration according to a gradient of the loss;updating an inverse mapping of the gradient of the potential function;creating, an updated neural network model using the network interface, the updated neural network model including an updated set of weights for a next time iteration according to the gradient of the potential function and the inverse mapping of the gradient of the potential function, wherein creating the updated neural network model bimodally concentrates a distribution of the updated set of weights; and storing the updated set of weights; andquantizing the trained neural network model after the plurality of time iterations, wherein quantizing the trained neural network model is performed according to a bimodality of the distribution of a trained set of weights.

176. The computing system of claim 175, wherein the trained neural network model is a multi-layered neural network model.

177. The computing system of claim 176, wherein the multi-layered neural network model is selected from the group consisting of a convolutional neural network model, a recursive neural network model, a multi-layer perceptron, and a generative adversarial network.

178. The computing system of claim 176, wherein the multi-layered neural network model is a large language model.

179. The computing system of any of claims 175-178, wherein the updated set of weights is further obtained based on at least one additional vector comprising a current set of auxiliary variables for the current neural network model.

180. The computing system of claim 179, wherein each sample of the set of training samples corresponds to a different auxiliary variable of the current set of auxiliary variables.

181. The computing system of claim 179 or claim 180, wherein obtaining the updated set of weights for the next time iteration comprises obtaining an updated set of auxiliary variables based on the current set of weights for the current time iteration.

182. The computing system of claim 181, wherein each auxiliary variable of the current set of auxiliary variables is obtained based on a regularization parameter and the loss function evaluated for the current set of weights at the current time iteration.

183. The computing system of claim 181 or claim 182, wherein a regularizer mirror descent (RMD) process is used to obtain at least one selected from the group consisting of the updated set of weights; and the current set of auxiliary variables.

184. The computing system of any of claims 181 -183, wherein iterative updates are used to obtain at least one selected from the group consisting of the updated set of weights; and the updated set of auxiliary variables.

185. The computing system of any of claims 179-184, wherein each of the current set of weights and the current set of auxiliary variables is initialized to at least one selected from the group consisting of: a zero vector and a non-zero vector comprising random numbers within a predetermined threshold of zero.

186. The computing system of any of claims 179-185, wherein the loss function comprises a constraint-enforcing loss computed from the current set of auxiliary variables.

187. The computing system of claim 186, wherein the constraint-enforcing loss is applied to prevent overfitting the neural network model to the set of training samples.

188. The computing system of any of claims 175-178, wherein a stochastic mirror descent (SMD) process is used to obtain the updated set of weights.

189. The computing system of any of claims 175-188, wherein the loss function includes a Kurtosis term.

190. The computing system of claim 189, wherein the Kurtosis term is a function of the current set of weights.

191. The computing system of any of claims 175-190, wherein supervised learning is used to train the neural network model.

192. The computing system of claim 191, wherein the loss function is configured to only equal zero when a predicted output value of the neural network model exactly matches a labeled output value from the set of training samples.

193. The computing system of any of claims 175-192, wherein the loss function is selected from the group consisting of a quadratic loss function and a cross-entropy loss function.

194. The computing system of any of claims 175-193, wherein, at the current time iteration, obtaining the updated set of weights is based, at least in part, on a learning rate at the current time iteration.

195. The computing system of claim 194, wherein the learning rate is time-invariant.

196. The computing system of any of claims 175-195, wherein updating the gradient of the potential function at the current time iteration is performed using at least one system selected from the group consisting of Adam, AdamW, AdaGrad, RMSProp, Momentum, and WeightDecay.

197. The computing system of any of claims 175-195, wherein the gradient of the potential function at the current time iteration is obtained recursively based on the gradient of the potential function at a previous time iteration.

198. The computing system of claim 197, wherein training the neural network model at the current time iteration further comprises storing the gradient of the potential function at the current time iteration.

199. The computing system of any of claims 175-198, wherein, after the plurality of time iterations, the potential function is minimized via implicit regularization.

200. The computing system of any of claims 175-199, wherein the potential function is a strictly convex function.

201. The computing system of claim 200, wherein the potential function is a norm function.

202. The computing system of claim 200 or claim 201, wherein the potential function is selected from the group consisting of:an £qnorm function where q » 1; andan £1+enorm function where 1 » E > 0.

203. The computing system of any of claims 175-202, wherein training the neural network model comprises at least one selected from the group consisting of distributed training, batch training, synchronous training, ensemble learning, meta learning, transfer learning, and continual learning.

204. The computing system of any of claims 175-203, wherein the inverse mapping of the gradient of the potential function is a many-to-one function.

205. The computing system of any of claims 175-204, wherein a data size of each individual weight of the current set of weights is quantized to a single bit.

206. The computing system of any of claims 175-205, wherein the potential function is selected to enable the bimodality of the distribution of the trained set of weights.

207. The computing system of claim 206, wherein the potential function is a rigid cage.

Images