Machine Learning Method for Sequence Prediction Using Sub-Sequence Weighting
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Recurrent neural networks (RNNs) face challenges in convergence and prediction time when dealing with sequence prediction problems, particularly for machine failure predictions, due to vanishing and exploding gradient issues, leading to uncertainty and long convergence times, which are unacceptable for real-world applications like machine failure prediction for large numbers of machines.
Innovation Solution
A hierarchical approach is used to determine weights for a target sequence by generating sub-sequences and determining initial weights based on averages of sub-sequences with fewer memory depth values, allowing for faster convergence without running RNN on the target sequence, and incorporating a high-dimensional projection process to reduce prediction time.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If standard RNN is used to model sequences for prediction, then the model can capture functional and temporal relationships, but convergence is not guaranteed and convergence time becomes excessively long
Solution Approach 1:
The patent segments the target sequence into multiple sub-sequences, each processed by a separate RNN instance. This division allows each RNN to converge faster on smaller subsequences while the ensemble of RNNs collectively captures the full temporal dependency of the original sequence, thus reducing overall convergence time while maintaining prediction accuracy.
Solution Approach 2:
The patent performs preliminary actions by pre-processing the target sequence into sub-sequences and training multiple RNNs on these smaller sequences before making predictions. This preliminary segmentation and distributed training approach enables faster convergence compared to training a single RNN on the complete long sequence.
2Reliability
If standard RNN is used for sequence prediction, then temporal dependencies can be modeled, but vanishing and exploding gradient problems occur leading to uncertainty in convergence
Solution Approach 1:
By segmenting the long target sequence into multiple shorter sub-sequences and training separate RNNs on each, the patent eliminates the vanishing and exploding gradient problems that occur in standard RNNs processing long sequences. Each RNN operates on a manageable sub-sequence where gradient flow is stable, ensuring reliable convergence without requiring complex architectural modifications.
Solution Approach 2:
The patent combines the outputs of multiple independent RNNs (each processing a sub-sequence) to form the final prediction. This merging approach allows each simple RNN to converge reliably on its sub-sequence while the ensemble collectively solves the original prediction problem, avoiding the need for complex specialized RNN structures.
3Reliability
If specialized RNN structures are used to overcome convergence problems, then some convergence issues are addressed, but convergence time remains too long for real-world prediction problems
Solution Approach 1:
The patent segments the prediction task into multiple parallel RNN instances, each handling a sub-sequence. This segmentation enables simultaneous processing of multiple subsequences, significantly increasing prediction throughput compared to sequential processing in specialized RNN structures, while maintaining convergence stability through the simpler architecture of individual RNNs.
Solution Approach 2:
Instead of using a single complex specialized RNN structure that processes the entire sequence sequentially, the patent employs multiple simpler RNNs that process sub-sequences in parallel. This partial action approach (multiple RNNs on parts of the sequence) exceeds the capability of a single RNN and achieves both fast convergence and high prediction throughput.
Data Source
AI summary
Embodiments of the invention provide machine learning method and system. The method comprises: generating a group of sub-sequences based on a target sequence including n basic memory depth values, the group of sub-sequences includes at least one subset of composite sequences, and each composite sequence in any subset is generated based on an equal number of consecutive basic memory depth values (BMDV); determining weights of each sub-sequence, wherein initial weights for a composite sequence generated based on m BMDV are determined based on average of weights of at least two sub-sequences each having an equal number of BMDV which is less than and closest to m; determining weights of the target sequence based on an average of weights of at least two sub-sequences each having an equal number of BMDV which is closest to n; and solving the prediction problem based on weights of the target sequence.