Neural Network Accelerator Local Memory Caching
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Neural network accelerators face latency issues when loading and switching between multiple models due to limited data transfer bandwidth between the host memory and the accelerator, which is problematic for real-time applications like object recognition and language translation.
Innovation Solution
Utilizing a local memory with a higher data transfer bandwidth to store and load neural network models, allowing for concurrent model loading and switching without interrupting computations, and sharing models among multiple accelerators to reduce latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If models are loaded from host memory to accelerator, then models can be stored in large capacity memory, but data transfer bandwidth is limited causing high latency
Solution Approach 1:
The patent divides the memory system into two segments: host memory for storing multiple models and local memory for caching active models. This segmentation allows the system to maintain large model storage capacity while providing fast access to currently used models through the local memory cache, thereby resolving the contradiction between storage capacity and access speed.
Solution Approach 2:
The local memory acts as an intermediary between the host memory and the neural network accelerator. It buffers model data transfers, allowing the accelerator to load models from the high-speed local memory rather than directly from the slower host memory, thus reducing loading latency while maintaining the ability to store multiple models in host memory.
2Adaptability or versatility
If multiple models are loaded and switched between accelerators, then application versatility is improved, but data transfer bandwidth limitations cause performance degradation
Solution Approach 1:
The system performs preliminary actions by pre-loading models into the local memory cache before they are needed by the accelerator. This allows models to be ready for immediate execution when switching between different neural network applications, maintaining versatility while avoiding performance degradation during model transitions.
Solution Approach 2:
The local memory enables continuous useful action by maintaining models in an easily accessible state between executions. Instead of repeatedly transferring models from host memory to accelerator, the local memory keeps models available for rapid switching, ensuring continuous high-performance processing across multiple model executions.
Data Source
AI summary
An integrated circuit device implementing a neural network accelerator may have a peripheral bus interface to interface with a host memory, and neural network models can be loaded from the host memory onto the state buffer of the neural network accelerator for execution by the array of processing elements. The neural network accelerator may also have a memory interface to interface with a local memory. The local memory may store neural network models from the host memory, and the models can be loaded from the local memory into the state buffer with reduced latency as compared to loading from the host memory. In systems with multiple accelerators, the models in the local memory can also be shared amongst different accelerators.