LSTM Inference Apparatus Switching Feedback Modes for Batch Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Inference processing using recurrent neural networks, specifically LSTM models, is hindered by the need for serial calculation due to two feedback loops, preventing the application of pipeline or parallel processing and thus making it difficult to reduce processing time.
Innovation Solution
An inference processing apparatus and method that switches between operation modes, alternately using and not using output feedback, allowing for batch processing of input data to reduce processing time by controlling the use of previous inference results and internal state values.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Measurement precision
If feedback loops are implemented in LSTM-based inference processing, then inference accuracy is improved, but processing speed deteriorates due to serial calculation requirement
Solution Approach 1:
The patent segments the feedback loop processing into multiple discrete stages: (1) obtaining current input data and previous inference results, (2) calculating current inference results using the LSTM model, (3) storing current inference results and previous internal states, (4) updating internal states. This segmentation allows for systematic optimization of each stage independently, enabling parallel processing possibilities while maintaining accuracy.
Solution Approach 2:
The patent dynamically adjusts the processing mode by switching between using feedback loops for accuracy-critical operations and bypassing feedback for speed-critical operations. The system determines whether to perform batch processing with feedback or rapid processing without feedback based on real-time requirements, optimizing the balance between accuracy and speed.
2Measurement precision
If feedback loops are used in LSTM inference, then model accuracy is maintained, but parallel processing cannot be applied
Solution Approach 1:
The patent implements periodic batch processing where feedback is applied at regular intervals rather than continuously at every time step. The system processes batches of data with feedback loops enabled, then switches to faster processing modes between batches, creating a periodic pattern that balances accuracy maintenance with throughput optimization.
Solution Approach 2:
The patent maintains continuous useful action by ensuring that even when feedback loops are temporarily disabled for speed optimization, the system continues processing data through alternative pathways. The previous inference results and internal states are preserved and reintegrated, ensuring no loss of accuracy while maintaining continuous productivity.
3Measurement precision
If serial calculation is performed to maintain feedback loop integrity, then inference accuracy is preserved, but processing time increases
Solution Approach 1:
The patent performs preliminary actions by pre-calculating and storing previous inference results and internal states in memory before they are needed for the next processing step. This preparation allows the feedback loop to be executed more efficiently when required, reducing the actual processing time while maintaining accuracy.
Solution Approach 2:
The patent implements skipping mechanisms where the system can bypass the feedback loop calculation for certain time steps or batches when accuracy requirements are temporarily reduced. This allows rapid processing through critical sections while still performing feedback calculations at key points to maintain overall inference accuracy.
Data Source
AI summary
The inference processing apparatus includes an inference calculator that performs calculation of a neural network based on input data xt of each consecutive time step and weights W of a trained neural network to infer features of the input data xt and also includes a memory that stores input data xt and weight W, a temporary memory that stores an output ht−1 of an inference result of an immediately previous time step, and a switching controller that controls switching between a first operation mode TM1 in which the inference calculator performs calculation of the neural network based on the input data xt, the weight W, and the output ht−1, at each time step and a second operation mode TM2 in which the inference calculator performs calculation of the neural network based on the input data xt and the weight W at each time step.