Stacked Auto Encoder Joint Optimization via ADMM
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current Stacked Auto Encoder (SAE) training methods face challenges in learning all parameters simultaneously due to the vanishing gradient problem, leading to sub-optimal greedy learning approaches where outer layers influence inner layers but not vice versa, making efficient regression training difficult.
Innovation Solution
A method is introduced that formulates regression into SAEs as a non-convex joint optimization function, reformulated as an Augmented Lagrangian formulation using Alternating Direction Method of Multipliers (ADMM), allowing simultaneous learning of encoder, decoder, and regression weights, enabling joint optimization of all layers.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If greedy layer-wise training is used to avoid vanishing gradient problem, then training stability is improved, but learning optimality deteriorates because outer layers influence inner layers but not vice versa
Solution Approach 1:
The patent segments the joint optimization problem into multiple sub-problems using ADMM, where each sub-problem corresponds to a specific layer or parameter set. This allows stable iterative optimization of each segment while converging to a globally optimal solution, resolving the contradiction between training stability and learning optimality.
Solution Approach 2:
Instead of the conventional forward layer-wise training approach, the patent inverts the optimization strategy by formulating a joint optimization function that considers all layers simultaneously, then solves it through backward iterative refinement using ADMM, enabling inner layers to influence outer layers and achieve optimal learning.
2Manufacturing precision
If all parameters are learned simultaneously through joint optimization, then learning optimality is improved, but computational complexity increases due to vanishing gradient problem
Solution Approach 1:
The patent divides the complex joint optimization problem into manageable sub-problems using ADMM, where each iteration optimizes a specific subset of parameters. This segmentation reduces computational complexity at each step while maintaining the benefit of simultaneous learning across all layers.
Solution Approach 2:
The patent performs preliminary formulation of the joint optimization function and identifies the vanishing gradient issue before training. By pre-processing the optimization problem and preparing the ADMM framework, the system enables efficient simultaneous parameter learning without being overwhelmed by computational complexity.
3Device complexity
If traditional layer-wise pre-training is used, then device complexity is reduced, but abstraction capacity deteriorates due to sub-optimal learning
Solution Approach 1:
The patent merges the training of all layers into a unified joint optimization framework, combining the learning processes of encoder layers, decoder layers, and regression parameters. This merging enables optimal abstraction capacity by allowing all layers to learn together while maintaining manageable complexity through ADMM.
Data Source
AI summary
A method and system for incorporating regression into a Stacked Auto Encoder utilizing deep learning based regression technique that enables joint learning of parameters for a regression model to train the SAE for a regression problem. The method comprises generating a regression model for the SAE for solving the regression problem, wherein regression model is formulated as a non-convex joint optimization function for an asymmetric SAE. The method further comprises reformulating the non-convex joint optimization function as an Augmented Lagrangian formulation in terms of a plurality of proxy variables and a plurality of hyper parameters. The method comprises splitting the Augmented Lagrangian formulation into sub-problems using Alternating Direction Method of Multipliers and jointly learning parameters for the regression model to train the SAE for the regression problem. The learned weights enable estimating the unknown target values.