Layer-Wise DNN Configuration Using Maximal Correlation Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing methods for training deep neural network (DNN) machine learning models, such as backpropagation, suffer from simultaneous and non-specific training of all layers, non-guaranteed optimality, slow convergence, and low explainability, due to the use of mean-square error (MSE) which is non-convex and poorly quantifies error statistics.
Innovation Solution
A modularized configuration framework for DNNs that individually configures each layer sequentially, using correlation measures based on maximal correlation concepts and least squares techniques to optimize parameters, ensuring transparency and improved convergence.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If backpropagation is used to train all layers simultaneously, then the training process can be implemented with standard methodologies, but the training convergence is slow and optimality is not guaranteed
Solution Approach 1:
The patent divides the simultaneous training of all layers into sequential training of individual layers. Each layer is trained independently in a specific sequence, with earlier layers fixed before training later layers. This segmentation resolves the contradiction by maintaining implementation simplicity while achieving faster convergence through layer-wise optimization rather than simultaneous updates.
Solution Approach 2:
The patent applies preliminary action by training and fixing earlier layers before training later layers. Each layer is configured in advance with its parameters optimized and fixed, providing a stable foundation for subsequent layers. This preliminary configuration enables faster overall convergence by avoiding the instability of simultaneous training updates.
2Ease of manufacture
If backpropagation with mean-square error is used, then the training process can be executed with standard algorithms, but the method introduces spurious correlations and provides poor proxy for optimality
Solution Approach 1:
The patent changes the fundamental parameter of the training objective from mean-square error to correlation ratio. This parameter change eliminates spurious correlations and provides a reliable proxy for optimality. The correlation ratio measures the actual functional relationship between layer outputs and targets, ensuring that training parameters are optimized for true predictive performance rather than artificial error minimization.
Solution Approach 2:
The patent substitutes the mechanical gradient descent optimization mechanism with a correlation-based optimization approach. Instead of relying on gradient calculations that can introduce spurious correlations, the system uses correlation ratio maximization to directly optimize the relationship between inputs and outputs, providing more reliable optimality guarantees.
3Productivity
If all layers are trained simultaneously with backpropagation, then the training process can be implemented efficiently, but the explainability of the model is reduced
Solution Approach 1:
The patent segments the training process into independent layer-wise training steps, where each layer's contribution can be individually analyzed and explained. This segmentation preserves explainability by allowing interpretation of each layer's specific transformation, while maintaining training efficiency through parallelizable layer configuration and fixed-parameter subsequent training.
Solution Approach 2:
The preliminary training and fixation of earlier layers creates a stable, interpretable foundation that preserves information about each layer's function. By fixing parameters after training, the system maintains clear causal relationships between layers, enabling better explanation of model behavior while keeping the overall training process efficient.
4Device complexity
If correlation measures are used to individually configure layers sequentially, then the computational complexity is reduced and explainability is enhanced, but the training process becomes more complex
Solution Approach 1:
The patent segments the training process into independent, manageable steps for each layer, reducing overall computational complexity. Each layer is trained separately with fixed parameters, allowing simpler computation at each step rather than complex simultaneous optimization. The modular structure makes the process easier to manufacture despite the increased number of training steps.
Solution Approach 2:
The parameter change from mean-square error to correlation ratio simplifies the optimization problem for each individual layer. The correlation ratio provides a direct measure of functional relationship that is easier to compute and optimize than gradient-based methods, reducing computational complexity while maintaining training effectiveness.
Data Source
AI summary
Various embodiments are directed to configuring or training deep neural network (DNN) machine learning models comprising one or more hidden layers and an output layer. Various embodiments provide technical advantages in training DNN machine learning models, including improved computational efficiency and guaranteed optimality. In one embodiment, an example method includes identifying a nonlinear-model-based representation for each hidden layer, which may be a Bank of Wiener Models, a nonlinear units of the hidden layer, and/or the like. The method further includes individually and sequentially configuring the hidden layers, each configured by determining a correlation measure (e.g., a correlation ratio) between the layer output and a target signal. Parameters of the particular hidden layer are modified by maximizing the correlation measure to yield maximal correlation over the space of functions. The method further includes performing automated tasks using the DNN machine learning model after configuring its parameters on a training set.