On-Chip Neural Network Weight Storage Architecture
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing neural network processing systems face performance bottlenecks due to the need to access off-chip memory for weight values, leading to high memory latency and reduced computational efficiency.
Innovation Solution
Storing weight values on-chip within the neural network processing engine, utilizing multiple memory banks that can be accessed independently to reduce memory latency and increase computational throughput.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If weight values are stored in off-chip memory, then device complexity is reduced, but memory latency increases and computational efficiency decreases
Solution Approach 1:
The patent divides the memory system into multiple on-chip memory banks that are spatially segmented and can be accessed independently. Each memory bank stores a portion of the weight values, allowing parallel access to different weight sets without interfering with each other. This segmentation reduces the time to retrieve weight values while maintaining manageable device complexity through modular memory organization.
Solution Approach 2:
The patent transitions from a single off-chip memory dimension to a multi-dimensional on-chip memory architecture where weight values are distributed across multiple memory banks in different spatial locations on the chip. This dimensional change enables simultaneous access to multiple weight values from different banks, dramatically reducing memory latency and improving computational throughput.
2Productivity
If weight values are stored on-chip in multiple memory banks, then memory latency is reduced and computational throughput increases, but device complexity increases
Solution Approach 1:
The on-chip memory banks are designed with multi-functionality, serving both as weight storage and as intermediate computation buffers. This universal design allows the same memory infrastructure to support multiple operations without requiring separate dedicated structures, thereby increasing computational throughput while limiting the growth of device complexity.
Solution Approach 2:
The patent merges the weight storage function with the computation buffer function into a unified on-chip memory system. By combining these functions into the same physical memory banks, the system achieves higher computational throughput without proportionally increasing device complexity, as the same hardware structure serves multiple purposes in the neural network processing pipeline.
3Speed
If multiple memory banks are used for weight storage, then memory access parallelism increases, but control complexity increases
Solution Approach 1:
The memory banks are designed with self-service capabilities, where each bank can be independently accessed and managed without requiring complex centralized control logic. The banks autonomously handle their own read/write operations, reducing the burden on the control unit and enabling faster memory access speeds with minimal control overhead.
Solution Approach 2:
The system performs preliminary actions by pre-loading weight values into the appropriate memory banks before computation begins. This advance organization of data in the multiple memory banks eliminates the need for complex real-time control logic during computation, as the data is already positioned for immediate access, thereby increasing memory access speed while keeping control complexity manageable.
Data Source
Figure 1
Figure 2A~2B
Figure 3A~3B
AI summary
Provided are systems, methods, and integrated circuits for neural network processing. In various implementations, an integrated circuit for neural network processing can include a plurality of memory banks storing weight values for a neural network. The memory banks can be on the same chip as an array of processing engines. Upon receiving input data, the circuit can be configured to use the set of weight values to perform a task defined for the neural network. Performing the task can include reading weight values from the memory banks, inputting the weight values into the array of processing engines, and computing a result using the array of processing engines, where the result corresponds to an outcome of performing the task.