NPU Feature Map Caching to Reduce Main-Memory Access
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural processing units (NPUs) face challenges with limited memory capacity, leading to increased data transfer and power consumption due to frequent feature map storage in main memory, and inefficient caching strategies that do not account for dynamic memory usage in neural networks.
Innovation Solution
A neural processing unit (NPU) that caches feature maps in a variable memory and reuses them using a delta-step caching algorithm, optimizing memory usage by determining caching entries based on space cost and caching profit, minimizing main memory access and overhead.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Quantity of substance
If feature maps are stored in main memory, then memory capacity is sufficient, but data transfer increases and power consumption increases
Solution Approach 1:
The patent divides the memory system into two segments: a small on-chip variable memory (buffer memory) and off-chip main memory. The variable memory is further segmented into multiple banks that can be independently managed. This segmentation allows frequently accessed feature maps to be cached in the small variable memory, reducing data transfer to main memory and lowering power consumption.
Solution Approach 2:
The patent implements a caching mechanism that pre-loads feature maps into the variable memory before they are needed for computation. The controller predicts which feature maps will be needed and loads them in advance, reducing the frequency of data transfer operations to main memory during actual computation, thereby reducing power consumption.
2Quantity of substance
If feature maps are stored in main memory, then memory capacity is sufficient, but data transfer increases and processing time increases
Solution Approach 1:
The patent divides the memory system into two segments: a small on-chip variable memory (buffer memory) and off-chip main memory. The variable memory is further segmented into multiple banks that can be independently managed. This segmentation allows frequently accessed feature maps to be cached in the small variable memory, reducing data transfer to main memory and lowering power consumption.
Solution Approach 2:
The patent implements a caching mechanism that pre-loads feature maps into the variable memory before they are needed for computation. The controller predicts which feature maps will be needed and loads them in advance, reducing the frequency of data transfer operations to main memory during actual computation, thereby reducing power consumption.
3Device complexity
If fixed caching strategy is used, then implementation is simple, but memory utilization is inefficient
Solution Approach 1:
The patent implements a dynamic caching strategy where the controller adaptively manages the variable memory based on runtime conditions. The controller determines which feature maps to cache, when to evict them, and how to allocate memory banks based on current computational needs and access patterns. This dynamic management maximizes memory utilization while maintaining reasonable implementation complexity.
Data Source
AI summary
A method of reusing feature maps that is operable on a neural processing unit (NPU) includes calculating, for a first layer of at least one artificial neural network (ANN) model, a space cost based on sizes of an input feature map, an output feature map, and a weight of a subsequent layer of the first layer; calculating a caching value for an operation of the first layer; determining a caching profit for the operation of the first layer based on the space cost and the caching value; determining a caching entry having maximum caching profit among caching candidate entries; and storing the caching entry in a variable memory. A system including the NPU includes a main memory to store a portion of the ANN model; a variable memory to selectively store a caching entry of the portion of the ANN model; and a controller to perform the method.