Derivative Neural Network Hardware Implementation for Dynamic RNNs
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current hardware systems are incapable of executing Recurrent Neural Networks (RNNs) due to their reliance on dynamic graphs, which cannot be represented as static graphs, limiting their ability to process sequences of indefinite length.
Innovation Solution
The method involves transforming the RNN into a derivative neural network by unrolling it over a predetermined number of steps, creating a static neural network that can be implemented on hardware capable only of feed-forward neural networks, where state outputs from one instance are used as inputs for subsequent instances, allowing operation over longer input sequences.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Adaptability or versatility
If the RNN is implemented using dynamic graphs, then the network can process sequences of indefinite length, but the hardware cannot execute the network
Solution Approach 1:
The RNN is segmented into multiple unrolled instances, each handling a fixed number of time steps. This segmentation transforms the dynamic graph into static subgraphs that can be executed on hardware, while maintaining the ability to process sequences of indefinite length by chaining multiple instances together.
Solution Approach 2:
The system maintains dynamic behavior through iterative application of static unrolled instances. By chaining multiple instances and passing state outputs from one instance to the next, the system achieves dynamic sequence processing capability on hardware that only supports static graph execution.
2Reliability
If the RNN is transformed into a static neural network by unrolling, then the hardware can execute the network, but the network can only process a predetermined number of inputs
Solution Approach 1:
The unrolled neural network instance is designed to be universal and reusable. By iterating the same unrolled instance multiple times and chaining them together, a single static network design can process sequences of any length, making the hardware implementation universally applicable to variable-length inputs.
Solution Approach 2:
The state outputs from each unrolled instance are continuously passed to the next instance, enabling uninterrupted processing across multiple instances. This continuous state transfer allows the system to process sequences longer than any single instance's predetermined length.
3Adaptability or versatility
If the RNN is unrolled over a predetermined plurality of steps, then the network becomes equivalent to a feed-forward network, but the device complexity increases
Solution Approach 1:
Instead of creating entirely separate hardware for each time step, the system uses copying by instantiating the same unrolled network design multiple times. This approach reduces device complexity by reusing the same hardware circuitry across multiple instances, rather than duplicating full RNN functionality for each step.
Data Source
AI summary
A method of implementing in hardware a recurrent neural network (RNN) for operation on a sequence of inputs, each step of the recurrent neural network being for operation on a different input of the sequence, the method comprising: receiving a representation of the RNN; transforming the representation of the RNN into a derivative neural network for operation over a predetermined plurality of inputs of the sequence of inputs, the derivative neural network having one or more state inputs and one or more state outputs and being equivalent to the RNN over a predetermined plurality of steps of the RNN; and iteratively applying the derivative neural network to the sequence of inputs by: implementing a sequence of instances of the derivative neural network in hardware; and providing the one or more state outputs from each instance of the derivative neural network at the hardware as the one or more state inputs to a subsequent instance of the derivative neural network at the hardware so as to operate the RNN over a sequence of inputs longer than the predetermined plurality of inputs.