Layer Trajectory LSTM With Depth Processing Block
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Deeply stacked Long Short-Term Memory (LSTM) models face training difficulties due to gradient vanishing issues and limited temporal modeling power, particularly in Automatic Speech Recognition (ASR) tasks, where future context frames carry valuable information for prediction.
Innovation Solution
The implementation of a layer trajectory LSTM model with a depth processing block that scans hidden states across time layers, incorporating future context frames through lookahead embedding or attention mechanisms to enhance classification accuracy and alleviate training challenges.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If LSTM models are stacked in many layers to improve modeling power, then the modeling capability is enhanced, but gradient vanishing issues occur and training becomes difficult
Solution Approach 1:
The patent introduces a depth processing block as an intermediary component that receives hidden states from multiple time layers and processes them together. This mediator allows information to flow across layers more effectively, preventing gradient vanishing while maintaining the benefits of deep stacking for improved modeling power.
Solution Approach 2:
The patent adds a depth dimension to the traditional temporal dimension of RNNs. By stacking LSTM layers in the depth dimension and using depth processing blocks to aggregate hidden states across layers, the model gains additional representational capacity without suffering from the same gradient vanishing issues as purely temporal extensions.
2Device complexity
If uni-directional RNN modeling is used to simplify the model, then the model complexity is reduced, but temporal modeling power is limited
Solution Approach 1:
The patent segments the temporal modeling task into multiple parallel time layers, each processing different temporal aspects. By dividing the temporal sequence into layered segments and processing them independently before aggregation, the model achieves enhanced temporal modeling power while keeping individual layer complexity manageable.
Solution Approach 2:
The patent merges the outputs of multiple uni-directional time layers through the depth processing block. By combining hidden states from parallel layers that each capture different temporal patterns, the model achieves superior temporal modeling power without requiring any single layer to be overly complex.
3Measurement precision
If future context frames are incorporated to improve prediction accuracy, then the classification accuracy is enhanced, but the computational complexity increases
Solution Approach 1:
The patent performs preliminary processing of future context frames by pre-computing their hidden states through the LSTM layers before they are needed for the current prediction. This allows the model to incorporate future information efficiently without recalculating during the prediction phase, reducing real-time computational complexity while maintaining accuracy.
Solution Approach 2:
The depth processing block serves multiple functions: it aggregates hidden states from current and future time layers, processes depth-wise connections, and outputs refined predictions. This multi-functional component reduces overall computational complexity by consolidating multiple operations into a single versatile block.
Data Source
AI summary
According to some embodiments, a machine learning model may include an input layer to receive an input signal as a series of frames representing handwriting data, speech data, audio data, and/or textual data. A plurality of time layers may be provided, and each time layer may comprise a uni-directional recurrent neural network processing block. A depth processing block may scan hidden states of the recurrent neural network processing block of each time layer, and the depth processing block may be associated with a first frame and receive context frame information of a sequence of one or more future frames relative to the first frame. An output layer may output a final classification as a classified posterior vector of the input signal. For example, the depth processing block may receive the context from information from an output of a time layer processing block or another depth processing block of the future frame.