Neural Accelerator Compiler for SRAM Model Reload Avoidance
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional hardware accelerators for neural networks require reloading of neural network models due to limited memory, causing significant latency and power consumption.
Innovation Solution
A compiler identifies frequently invoked neural network models and compiles them in a sequence, storing the compiled models in static random access memory (SRAM) with a unique hash to avoid redundant reloading, allowing simultaneous storage and retrieval.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If the hardware accelerator memory stores parameters of only one neural network model, then the memory capacity is sufficient, but the latency increases due to repeated reloading of model parameters
Solution Approach 1:
The compiler performs preliminary compilation and loading of multiple frequently invoked neural network models into the hardware accelerator memory in advance. By predicting which models will be needed and pre-loading them, the system avoids the time-consuming process of reloading models during execution, thus reducing latency while maintaining sufficient memory capacity
Solution Approach 2:
The system dynamically manages memory allocation by tracking the invocation frequency of different neural network models. The compiler identifies frequently invoked models and prioritizes their loading into memory, while less frequently used models are kept outside memory. This dynamic approach allows the limited memory capacity to be used more effectively, storing the right models at the right time to minimize latency
2Adaptability or versatility
If the hardware accelerator memory is cleared to load a new neural network model, then the new model can be executed, but power consumption increases due to repeated memory operations
Solution Approach 1:
The compiler performs preliminary analysis of model invocation patterns and pre-loads frequently used models into memory before they are needed. This eliminates the need for repeated memory clearing and reloading operations, significantly reducing power consumption while maintaining the ability to execute different models as required
Solution Approach 2:
The system maintains continuous useful action by keeping frequently invoked neural network models loaded in memory across multiple execution cycles. Instead of clearing and reloading memory with each model execution, the system continuously retains the necessary models in memory, thereby eliminating redundant memory operations and reducing overall power consumption
Data Source
AI summary
A compiler of a computing device is described that identifies a sequence of neural network models frequently invoked by an application of the computing device, compiles the models in that sequence, and loads a static random access memory (SRAM) of a hardware accelerator with the compiled models only when the same compiled models—from another, but same, sequence that was previously invoked—are not already present in the SRAM. This prevents unnecessary reloading of compiled models into the SRAM, thereby increasing runtime speed and conserving computational energy.