Deep Convex Network Parallel Training via Convex Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Training deep belief networks (DBNs) for speech recognition is challenging due to the difficulty in parallelizing conventional stochastic gradient descent learning algorithms across machines, making the process tedious and inefficient.
Innovation Solution
A deep convex network is introduced, comprising layered modules with a first linear layer, a non-linear hidden layer, and a second linear layer, where the second weight matrix is learned using convex optimization, enabling batch-based parallelization and efficient training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If stochastic gradient descent learning algorithm is used to train DBNs, then learning accuracy can be improved, but parallelization across machines becomes extremely difficult
Solution Approach 1:
The training process is segmented into two distinct phases: pre-training phase using restricted boltzmann machines for each layer, and fine-tuning phase using stochastic gradient descent. This segmentation allows the computationally intensive parallelizable pre-training to be distributed across multiple machines while reserving the sequential fine-tuning for final accuracy optimization.
Solution Approach 2:
The restricted boltzmann machine pre-training is performed as a preliminary action before applying stochastic gradient descent. This preliminary phase initializes weight matrices in a parallelizable manner, creating a good starting point that reduces the burden on subsequent sequential fine-tuning while enabling efficient distributed computation.
2Reliability
If conventional stochastic gradient descent is used for DBN training, then learning effectiveness is maintained, but training efficiency and speed deteriorate due to difficulty in parallelization
Solution Approach 1:
The training process is divided into pre-training (parallelizable, efficiency-focused) and fine-tuning (sequential, effectiveness-focused) stages. This segmentation allows maximizing training efficiency through parallel distributed computation in the pre-training phase while preserving learning effectiveness through careful fine-tuning in the second phase.
Solution Approach 2:
The training alternates between parallel pre-training iterations and sequential fine-tuning iterations. This periodic switching between parallel and sequential modes allows the system to periodically refresh weights through parallel computation while maintaining effectiveness through periodic fine-tuning updates.
Data Source
AI summary
A method is disclosed herein that includes an act of causing a processor to access a deep-structured, layered or hierarchical model, called a deep convex network, retained in a computer-readable medium, wherein the deep-structured model comprises a plurality of layers with weights assigned thereto. This layered model can produce the output serving as the scores to combine with transition probabilities between states in a hidden Markov model and language model scores to form a full speech recognizer. Batch-based, convex optimization is performed to learn a portion of the deep convex network's weights, rendering it appropriate for parallel computation to accomplish the training. The method can further include the act of jointly substantially optimizing the weights, the transition probabilities, and the language model scores of the deep-structured model using the optimization criterion based on a sequence rather than a set of unrelated frames.