RNN Number Format Selection for Hardware Accelerator Execution
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware accelerators struggle to efficiently execute Recurrent Neural Networks (RNNs) due to their dynamic nature, leading to poor performance, low utilization of parallel architectures, and high memory bandwidth consumption.
Innovation Solution
Transform RNNs into static neural networks by unrolling them over a predetermined number of timesteps, separating causal and non-causal operations, and implementing them on hardware accelerators using a transformation unit and control logic to optimize tensor operations.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If RNNs are executed directly on hardware accelerators, then the dynamic nature of RNNs is preserved, but performance is poor and memory bandwidth consumption is high
Solution Approach 1:
The RNN execution is segmented into multiple static time steps, where each time step is processed independently by the hardware accelerator. This segmentation allows the accelerator to optimize for each static segment while maintaining the overall dynamic behavior of the RNN through sequential processing of segments.
Solution Approach 2:
The system dynamically determines the number of time steps to unroll based on input sequence characteristics, and dynamically configures hardware resources for each time step. This allows the static hardware architecture to adapt to dynamic RNN workloads while maintaining optimization benefits.
2Productivity
If RNNs are unrolled over predetermined timesteps, then parallel processing capabilities are maximized, but memory bandwidth requirements increase
Solution Approach 1:
Data is pre-processed and organized into batches before being fed to the accelerator, and intermediate results are cached between time steps. This preliminary preparation reduces the frequency and volume of memory accesses during actual computation, lowering overall memory bandwidth consumption while enabling parallel processing.
Solution Approach 2:
Multiple time steps are merged into a single parallel computation graph that can be executed simultaneously on the hardware accelerator. By combining operations across time steps and identifying shared computations, the system achieves parallel processing efficiency while reducing redundant memory accesses.
3Productivity
If RNNs are transformed into static neural networks, then hardware accelerator utilization is improved, but the dynamic behavior of RNNs is lost
Solution Approach 1:
The system applies periodic boundary conditions by using the output state of the final time step as the initial state for the next batch of inputs. This periodic state transfer preserves the dynamic recurrent behavior across batches while allowing each batch to be processed as a static computation graph optimized for hardware acceleration.
Solution Approach 2:
A state buffer acts as an intermediary between static time step computations, carrying the recurrent state information forward. This intermediary mechanism allows the system to maintain dynamic RNN behavior through state persistence while each individual time step remains a static, hardware-optimizable computation.
Data Source
Figure 1~2
Figure 3
Figure 4
AI summary
A computer-implemented method of selecting a number format for representing two or more values of a recurrent neural network (RNN) for use in configuring a hardware implementation of the RNN, the method comprising: receiving a representation of the RNN; implementing the representation of the RNN as a test neural network for operation on a sequence of test inputs, each step of the test neural network comprising an instance of the two or more values of the RNN; operating the test neural network for a plurality of steps on the sequence of test inputs and collecting statistics for provision to a number format selection algorithm; and applying a number format selection algorithm to the statistics so as to derive a common number format for the plurality of instances of the two or more values of the RNN.