RNN Training Apparatus Variable-Length Chunk Segmentation
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The training of recurrent neural networks (RNNs) using truncated back propagation through time (TBPTT) faces challenges in maintaining continuity of the hidden state, leading to reduced freedom in selecting time-series data for mini-batches, data bias, unstable convergence, and decreased training efficiency due to the need for shuffling blocks of fixed length.
Innovation Solution
An RNN training apparatus that includes a storage unit for intermediate output data, a construction unit for selecting sequences to form mini-batches, a reading unit for unprocessed hidden states, and an optimization unit for calculations, allowing for flexible mini-batch construction and reduced padding, thereby maintaining hidden state continuity and improving training efficiency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If blocks of fixed length are divided for TBPTT training, then error back propagation is enabled in units of blocks, but continuity of hidden state is interrupted and degree of freedom in selecting time-series data is reduced
Solution Approach 1:
The patent segments the time-series data into variable-length chunks rather than fixed-length blocks. Each chunk contains a variable number of time steps based on the actual sequence length, allowing flexible combination of chunks while maintaining hidden state continuity. This resolves the contradiction by enabling data selection freedom through variable segmentation rather than rigid fixed-length division.
Solution Approach 2:
The patent introduces dynamic mini-batch construction where the composition of mini-batches can change freely based on data characteristics rather than being constrained by fixed block divisions. The system dynamically selects and combines variable-length chunks to form mini-batches, maintaining hidden state continuity while maximizing data selection flexibility and training efficiency.
2Reliability
If shuffling is performed on blocks divided by TBPTT, then data bias is avoided, but continuity of hidden state is interrupted and training efficiency is deteriorated
Solution Approach 1:
The patent segments data into variable-length chunks that preserve temporal continuity, then combines these chunks into mini-batches without requiring traditional shuffling operations. This segmentation approach maintains hidden state continuity while still achieving data diversity through flexible chunk combination, eliminating the need for shuffling that would interrupt continuity.
Solution Approach 2:
The patent changes the parameter of data division from fixed-length blocks to variable-length chunks. This parameter change allows the system to maintain hidden state continuity while achieving data bias avoidance through different means (variable-length random sampling and flexible mini-batch construction), thereby improving training efficiency without sacrificing convergence stability.
3Reliability
If fixed-length blocks are used for mini-batch construction, then TBPTT error back propagation is enabled, but data bias increases and convergence becomes unstable
Solution Approach 1:
The patent segments time-series data into variable-length chunks that adapt to actual sequence lengths. This segmentation enables flexible mini-batch construction that avoids the data bias inherent in fixed-length blocking. By combining variable-length chunks, the system achieves both convergence stability and reduced data bias, eliminating the harmful effects of fixed-length block constraints.
Solution Approach 2:
The patent introduces dynamic mini-batch construction methods that adapt to data characteristics rather than following fixed patterns. The system dynamically selects and combines variable-length chunks to form mini-batches, eliminating the rigid fixed-length block structure that causes data bias. This dynamic approach stabilizes convergence while reducing harmful data bias effects.
Data Source
AI summary
The RNN training apparatus includes a storage and processing circuitry. The storage stores a hidden state of an RNN for the N sequences. The processing circuitry selects M (M<N) sequences from N sequences used for training of the RNN to construct a mini-batch and outputs sequence information identifying the selected sequence. The processing circuitry reads the unprocessed hidden state of the sequence corresponding to the sequence information from the storage according to the sequence information. The processing circuitry optimize the RNN based on the unprocessed hidden state and the mini-batch. The processing circuitry writes the processed hidden state obtained by the optimization, in the storage according to the sequence information.