Neural Network Model Sequencing in SRAM to Avoid Reload Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional hardware accelerators face significant latency and power consumption due to the need to reload and recompile neural network models frequently, as they can only store parameters for a single model at a time, leading to inefficient memory usage and processing delays.
Innovation Solution
A compiler identifies frequently invoked neural network models and compiles them in a sequence, storing the compiled models in SRAM only when the same sequence is detected, using a hash system to prevent unnecessary reloading and clearing, allowing simultaneous storage and retrieval of multiple models for faster processing.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If parameters of a single neural network model are stored in the memory of the hardware accelerator, then the model can be executed, but when another model needs to be executed, parameters must be reloaded causing latency and power consumption
Solution Approach 1:
The system pre-loads parameters of frequently invoked neural network models into the hardware accelerator's memory in advance, so that when a model is requested, its parameters are already available, eliminating the need for reloading and thus reducing latency
Solution Approach 2:
The system dynamically manages the set of pre-loaded model parameters based on invocation frequency, using a compiler to identify frequently invoked models and updating the pre-loaded set as usage patterns change, optimizing the balance between memory usage and processing speed
2Productivity
If parameters of frequently invoked neural network models are pre-loaded into the memory of the hardware accelerator, then processing speed is improved, but memory resources are consumed
Solution Approach 1:
The system loads parameters of only the most frequently invoked models into the hardware accelerator's memory, rather than loading all possible models, thus using a partial set of available memory resources to achieve the greatest performance benefit
Solution Approach 2:
The system changes the parameter of invocation frequency to determine which models to pre-load, using a compiler to analyze and identify models with high invocation frequency, thus dynamically adjusting which parameters are stored in memory based on usage patterns
3Device complexity
If the hardware accelerator stores parameters for only one neural network model at a time, then memory management is simple, but reloading parameters for different models causes significant power consumption
Solution Approach 1:
The system pre-loads parameters of multiple frequently invoked models into the hardware accelerator's memory in advance, so that when a model is requested, its parameters are already available, eliminating the need for reloading and thus reducing power consumption
Solution Approach 2:
The system dynamically manages the set of pre-loaded model parameters based on invocation frequency, using a compiler to identify frequently invoked models and updating the pre-loaded set as usage patterns change, optimizing the balance between memory usage and processing speed
Data Source
Figure 1
Figure 2
Figure 3
AI summary
A compiler of a computing device is described that identifies a sequence of neural network models frequently invoked by an application of the computing device, compiles the models in that sequence, and loads a static random access memory (SRAM) of a hardware accelerator with the compiled models only when the same compiled models-from another, but same, sequence that was previously invoked-are not already present in the SRAM. This prevents unnecessary reloading of compiled models into the SRAM, thereby increasing runtime speed and conserving computational energy.