Neural Network Accelerator Layer Scheduling via Shared Memory
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
The transfer of data between a host computer system and a neural network accelerator in hardware accelerator-based implementations of neural networks can degrade performance due to the need for frequent data transfers, which is inefficient compared to CPU or GPU implementations.
Innovation Solution
A method where the host computer system copies an input data matrix to shared memory and instructs the neural network accelerator to perform operations on each layer sequentially, waiting for completion before initiating the next layer, and using the results data matrix from one layer as input for the next, thereby reducing the need for time-consuming data transfers between the host and accelerator.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Power
If data is transferred between host computer system and neural network accelerator for each layer operation, then the neural network can be processed by hardware accelerator with faster speed and lower power consumption, but the frequent data transfers degrade overall performance
Solution Approach 1:
The patent merges the neural network accelerator with shared memory into a unified system where the accelerator can directly access and operate on data in the shared memory without requiring transfers to host memory. This integration eliminates the performance-degrading data transfer step while maintaining the power efficiency benefits of hardware acceleration.
Solution Approach 2:
The shared memory acts as an intermediary between the host computer system and the neural network accelerator, allowing the accelerator to access input data and store results without direct host intervention. This mediator enables the accelerator to perform multiple layer operations autonomously, reducing the frequency of data transfers and improving overall processing performance.
2Speed
If data is transferred between host computer system and neural network accelerator for each layer operation, then the neural network can be processed by hardware accelerator with faster speed, but the frequent data transfers degrade overall performance
Solution Approach 1:
The patent merges the neural network accelerator with shared memory into a unified system where the accelerator can directly access and operate on data in the shared memory without requiring transfers to host memory. This integration eliminates the performance-degrading data transfer step while maintaining the speed advantages of hardware acceleration.
Solution Approach 2:
The shared memory enables continuous operation of the neural network accelerator by providing persistent access to data across multiple layer operations. The accelerator can continuously process data through multiple layers without interruption for data transfers, maintaining high processing speed and improving overall productivity.
3Productivity
If host waits for completion of each layer before initiating the next layer, then data transfer overhead is minimized, but control logic complexity increases
Solution Approach 1:
The neural network accelerator is designed to autonomously manage its own operation sequence, automatically waiting for completion of one layer before initiating the next layer without requiring complex host control logic. The accelerator self-manages the layer-by-layer processing workflow, reducing control complexity while maintaining processing efficiency.
Solution Approach 2:
The system implements a feedback mechanism where the neural network accelerator signals completion of each layer operation, and this completion signal automatically triggers the initiation of the next layer. This feedback-based control simplifies the overall control logic by using straightforward signal-based coordination between host and accelerator rather than complex sequential control.
Data Source
AI summary
In disclosed approaches of neural network processing, a host computer system copies an input data matrix from host memory to a shared memory for performing neural network operations of a first layer of a neural network by a neural network accelerator. The host instructs the neural network accelerator to perform neural network operations of each layer of the neural network beginning with the input data matrix. The neural network accelerator performs neural network operations of each layer in response to the instruction from the host. The host waits until the neural network accelerator signals completion of performing neural network operations of layer i before instructing the neural network accelerator to commence performing neural network operations of layer i+1, for i≥1. The host instructs the neural network accelerator to use a results data matrix in the shared memory from layer i as an input data matrix for layer i+1 for i≥1.