Neural Processing Device Dynamic Memory Bandwidth Control
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural processing units (NPUs) face performance issues due to increased memory bank conflicts as the number of memory clients grows, leading to inefficiencies in memory usage and increased power/area overhead.
Innovation Solution
A neural processing device with a first memory for universal data and a second memory with a lower capacity, along with a bandwidth control path and control logic that dynamically reconfigures memory bandwidth and storage location based on the layer configuration of an artificial neural network, to optimize memory usage and prevent conflicts.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the number of memory banks is increased to prevent memory bank conflicts, then memory bank conflict is reduced, but power and memory area overhead increase
Solution Approach 1:
The patent implements dynamic memory bank selection where the system can switch between different memory banks based on real-time access patterns and conflict detection. The memory controller dynamically routes memory requests to appropriate banks, allowing the system to use fewer physical banks while maintaining low conflict rates through intelligent scheduling rather than simply increasing the number of banks statically.
Solution Approach 2:
The system changes operational parameters by implementing variable memory bank allocation where the number of active banks can be adjusted based on workload characteristics. The patent uses parameters such as access patterns, data sizes, and temporal locality to determine optimal bank allocation, allowing the system to adapt the memory architecture behavior without physical reconfiguration.
2Reliability
If the number of memory banks is increased to prevent memory bank conflicts, then memory bank conflict is reduced, but memory area overhead increases
Solution Approach 1:
The patent implements dynamic memory bank selection where the system can switch between different memory banks based on real-time access patterns and conflict detection. The memory controller dynamically routes memory requests to appropriate banks, allowing the system to use fewer physical banks while maintaining low conflict rates through intelligent scheduling rather than simply increasing the number of banks statically.
Solution Approach 2:
The memory banks are designed with multi-functional capabilities where each bank can serve multiple memory clients through time-division and priority-based access. The patent implements a universal memory access mechanism that allows any bank to handle any client's data based on current system state, reducing the need for dedicated banks per client and thereby reducing total area requirements.
3Device complexity
If a single integrated memory is used, then device complexity is reduced, but memory bank conflict increases when the number of memory clients increases
Solution Approach 1:
The patent segments the memory access control functionality rather than physically segmenting the memory structure. It divides the memory bandwidth and access rights into multiple controllable channels managed by the memory controller, allowing virtual separation of memory paths for different clients without creating separate physical memory banks. This maintains structural simplicity while reducing effective conflicts.
Solution Approach 2:
The memory controller acts as an intermediary between memory clients and the memory array. It mediates access conflicts by implementing arbitration logic, priority management, and bank routing functions. This intermediary layer resolves conflicts before they reach the memory array, allowing a simple integrated memory structure to serve multiple clients without direct bank conflicts.
4Ease of manufacture
If memory size of a memory bank is fixed, then manufacturing is simplified, but memory usage becomes limited and ineffective generating area overhead
Solution Approach 1:
The patent implements dynamic memory bank selection where the system can switch between different memory banks based on real-time access patterns and conflict detection. The memory controller dynamically routes memory requests to appropriate banks, allowing the system to use fewer physical banks while maintaining low conflict rates through intelligent scheduling rather than simply increasing the number of banks statically.
Solution Approach 2:
The system changes operational parameters by implementing variable memory bank allocation where the number of active banks can be adjusted based on workload characteristics. The patent uses parameters such as access patterns, data sizes, and temporal locality to determine optimal bank allocation, allowing the system to adapt the memory architecture behavior without physical reconfiguration.
Data Source
AI summary
A neural processing device includes a first memory configured to store universal data, a second memory distinguished from the first memory and having a capacity less than that of the first memory, a bandwidth control path configured to reconfigure a memory bandwidth for memory clients to use one of the first memory and the second memory based on a control signal, and a control logic configured to calculate a target capacity for data of a target client of the memory clients determined based on a layer configuration of an artificial neural network, and generate the control signal to store the data of the target client in the second memory based on a result of comparing the target capacity and the capacity of the second memory.