Deep Belief Network Training via Sequence-Level Optimization
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep belief networks (DBNs) face challenges in learning due to the increased likelihood of getting trapped in local optima, especially with back-propagation algorithms not performing effectively, which hinders their modeling power in speech recognition tasks.
Innovation Solution
A pretraining procedure is implemented using Restricted Boltzmann Machines (RBM) to learn initial weights between layers, followed by joint optimization of DBN weights, transition parameters, and language model scores through discriminative training at the sequence level, addressing the limitations of frame-level information training.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of manufacture
If back-propagation algorithm is used for training DBN, then training process is simple, but training effectiveness deteriorates due to trapping in local optima
Solution Approach 1:
The patent applies preliminary action by using a greedy layer-wise training approach to pre-train each layer of the DBN before full sequence training. This preliminary training establishes reasonable initial weights that prevent the back-propagation algorithm from getting trapped in poor local optima during subsequent optimization, thereby maintaining both training simplicity and effectiveness.
Solution Approach 2:
The patent changes training parameters by transitioning from frame-level information to full-sequence information, and from layer-by-layer greedy training to joint optimization. This parameter change allows the system to escape local optima while maintaining training feasibility through structured optimization approaches.
2Device complexity
If frame-level information is used for training, then training complexity is low, but recognition accuracy deteriorates
Solution Approach 1:
The patent merges frame-level information with full-sequence information in a unified training framework. By combining local frame-level features with global sequence context, the system achieves high recognition accuracy without excessive training complexity, as the merged information is processed through efficient gradient-based optimization.
Solution Approach 2:
The patent transitions from one-dimensional frame-level training to multi-dimensional full-sequence training by incorporating temporal dependencies across the entire speech sequence. This dimensional expansion captures higher-order correlations while maintaining computational efficiency through structured optimization methods.
3Adaptability or versatility
If DBN with many hidden layers is used, then modeling power increases, but learning difficulty increases due to local optimum trapping
Solution Approach 1:
The patent applies preliminary action by using greedy layer-wise pre-training to establish reasonable initial weights for each layer before full sequence training. This preliminary step prevents deep networks from getting trapped in poor local optima during subsequent optimization, enabling effective learning of high-order correlations without excessive difficulty.
Solution Approach 2:
The patent implements feedback mechanisms through joint optimization where training signals propagate through all layers simultaneously. This feedback allows each layer to adjust its parameters based on global performance metrics, preventing local optima trapping while maintaining the benefits of deep architecture for capturing complex patterns.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A method is disclosed herein that include an act of causing a processor to access a deep-structured model retained in a computer-readable medium, wherein the deep-structured model comprises a plurality of layers with weights assigned thereto, transition probabilities between states, and language model scores. The method can further include the act of jointly substantially optimizing the weights, the transition probabilities, and the language model scores of the deep-structured model using the optimization criterion based on a sequence rather than a set of unrelated frames.