Machine Learning Models With Depth Processing Units for Stable Training
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deep recurrent neural networks, such as LSTM models, face challenges with training due to the gradient vanishing issue when layered excessively, and existing solutions like skip connections do not fully address these issues, leading to inefficiencies in training and performance.
Innovation Solution
Implementing a layer processing block between LSTM layers that scans outputs from time layers for summarized information, using parallel threads for forward-propagation, and incorporating an attention layer for enhanced classification accuracy.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If LSTM models are stacked in too many layers to get better modeling power, then the modeling power is improved, but the gradient vanishing issue occurs and training becomes very hard
Solution Approach 1:
The patent introduces depth processing units as intermediary components between LSTM layers. These depth processing units process features from multiple LSTM layers simultaneously, acting as mediators that aggregate information across layers without requiring excessive layer stacking, thereby maintaining training stability while improving modeling power.
Solution Approach 2:
The patent adds a depth dimension to the traditional layer-wise processing by introducing depth processing units that operate across multiple LSTM layers in parallel. This dimensional change allows the model to capture hierarchical features more efficiently without increasing the number of sequential LSTM layers, thus avoiding gradient vanishing while enhancing modeling capability.
2Reliability
If skip connections are added between layers to solve gradient vanishing, then training stability is improved, but device complexity increases
Solution Approach 1:
The patent merges the functionality of skip connections into the depth processing units, which simultaneously perform feature aggregation from multiple layers and depth-wise processing. This consolidation reduces the number of separate components compared to traditional skip connection architectures, thereby improving training stability while controlling structural complexity.
3Measurement precision
If more LSTM layers are added to improve classification accuracy, then classification accuracy is improved, but computational overhead increases
Solution Approach 1:
The patent introduces a depth dimension that processes features from multiple LSTM layers in parallel through depth processing units. This allows the model to achieve higher classification accuracy by leveraging information from multiple layers simultaneously without sequentially processing through excessive LSTM layers, thereby reducing computational overhead while maintaining or improving accuracy.
Solution Approach 2:
The depth processing units selectively process features from specific LSTM layers based on their relevance to the classification task, rather than uniformly processing all layers. This partial processing approach reduces unnecessary computational overhead while maintaining classification accuracy by focusing computational resources on the most informative features.
Data Source
Figure 1
Figure 2
Figure 3
AI summary
Representative embodiments disclose machine learning classifiers used in scenarios such as speech recognition, image captioning, machine translation, or other sequence-to-sequence embodiments. The machine learning classifiers have a plurality of time layers, each layer having a time processing block and a depth processing block. The time processing block is a recurrent neural network such as a Long Short Term Memory (LSTM) network. The depth processing blocks can be an LSTM network, a gated Deep Neural Network (DNN) or a maxout DNN. The depth processing blocks account for the hidden states of each time layer and uses summarized layer information for final input signal feature classification. An attention layer can also be used between the top depth processing block and the output layer.