On-Chip Memory Banks for Low-Weight-Reuse Neural Network Processing
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network processors are limited by memory bandwidth due to weight values being stored off-chip, leading to suboptimal performance and response times, especially in deep neural networks with low weight reuse factors.
Innovation Solution
Storing weight values on-chip within a neural network processor using multiple independently accessible memory banks, allowing simultaneous access and reducing memory latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Speed
If weight values are stored off-chip in existing neural network processors, then device complexity is reduced, but memory bandwidth becomes limited and response time increases
Solution Approach 1:
The patent divides the memory system into multiple independently accessible memory banks on-chip, allowing parallel access to different weight values. This segmentation enables simultaneous reading of multiple weights during neural network computation, dramatically improving memory bandwidth and reducing response time compared to off-chip storage.
Solution Approach 2:
The patent transitions from a single-dimensional off-chip memory access model to a multi-dimensional on-chip memory bank structure. By organizing weights across multiple memory banks that can be accessed in parallel, the system adds a dimension of concurrency to memory access, effectively increasing bandwidth without proportionally increasing complexity.
2Productivity
If weight values are stored on-chip in multiple memory banks, then memory bandwidth and computational efficiency are improved, but device complexity increases
Solution Approach 1:
By segmenting the weight storage into multiple independent memory banks, the system enables parallel access patterns that match the parallel nature of neural network computations. Each memory bank can serve different computation units simultaneously, improving productivity while keeping individual bank complexity manageable.
Solution Approach 2:
The memory banks are designed to be universally accessible by multiple computation units, allowing the same memory infrastructure to support various neural network operations and configurations. This multi-functionality reduces the need for dedicated memory for each computation unit, balancing productivity gains with complexity control.
3Loss of time
If weight values are stored off-chip, then ease of manufacture is improved, but memory latency increases leading to suboptimal performance
Solution Approach 1:
The patent implements multiple smaller memory banks on-chip rather than a single large off-chip memory. This segmentation reduces the latency of accessing individual weight values while distributing the manufacturing complexity across standard on-chip memory structures that are easier to integrate than large off-chip packages.
Solution Approach 2:
Weight values are pre-loaded into on-chip memory banks before computation begins. This preliminary action eliminates the need for real-time off-chip memory access during computation, reducing latency while allowing weights to be loaded during chip initialization or training phases when the system is already powered on.
Data Source
AI summary
Provided are systems, methods, and integrated circuits for neural network processing. In various implementations, an integrated circuit for neural network processing can include a plurality of memory banks storing weight values for a neural network. The memory banks can be on the same chip as an array of processing engines. Upon receiving input data, the circuit can be configured to use the set of weight values to perform a task defined for the neural network. Performing the task can include reading weight values from the memory banks, inputting the weight values into the array of processing engines, and computing a result using the array of processing engines, where the result corresponds to an outcome of performing the task.