Deep Learning Accelerator Caching via Dynamic Hint-Based Bypass
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Current systems for processing artificial neural networks (ANNs) face challenges in reducing energy consumption and computation time, particularly due to limitations in memory access and caching strategies, which hinder efficient performance of deep learning accelerators.
Innovation Solution
The implementation of a deep learning accelerator (DLA) with local memory and a system buffer that allows for selective caching and bypassing of the system buffer based on data type and priority hints, optimizing data access and reducing memory bandwidth usage.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If data is cached in the system buffer for all operations, then data access speed is improved, but memory bandwidth usage increases and power consumption increases
Solution Approach 1:
The system dynamically changes the caching parameter (whether to cache data in system buffer) based on data type and reuse probability, rather than using a fixed caching policy. This allows the system to optimize between access speed and power consumption by selectively caching only beneficial data
Solution Approach 2:
The caching behavior is made dynamic through the use of hints from deep learning instructions that indicate data type and expected reuse. The system adapts its caching strategy in real-time based on these hints, rather than following a static caching policy
2Speed
If data is cached in the system buffer, then data access speed is improved, but memory bandwidth usage increases
Solution Approach 1:
The system changes the caching parameter based on data characteristics indicated by hints, optimizing memory bandwidth usage by caching only data that will be reused, rather than caching all data uniformly
3Productivity
If the system buffer is used for caching, then computing performance is improved, but device complexity increases
Solution Approach 1:
The hint mechanism acts as an intermediary that provides information about data characteristics to the caching system. This simple intermediary enables sophisticated selective caching without requiring complex analysis of data access patterns
Solution Approach 2:
The caching system is segmented into different paths: one for data that should be cached (based on hints indicating high reuse probability) and one for data that should bypass the cache. This segmentation simplifies the decision logic compared to a unified complex caching policy
Data Source
AI summary
Systems, devices, and methods related to a deep learning accelerator and memory are described. For example, the accelerator can have processing units to perform at least matrix computations of an artificial neural network via execution of instructions. The processing units have a local memory store operands of the instructions. The accelerator can access a random access memory via a system buffer, or without going through the system buffer. A fetch instruction can request an item, available at a memory address in the random access memory, to be loaded into the local memory at a local address. The fetch instruction can include a hint for the caching of the item in the system buffer. During execution of the instruction, the hint can be used to determine whether to load the item through the system buffer or to bypass the system buffer in loading the item.