Memory Bank Buffers for Neural Network Read-Modify-Write
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing memory systems face inefficiencies in processing read-modify-write requests, particularly in neural networks where residual blocks require element-wise operations between tensors stored in different memory locations.
Innovation Solution
A memory system architecture that processes read-modify-write requests by routing them to appropriate memory banks, using bank-specific buffers and combiner circuits for element-wise operations, and applying activation functions to the results before writing them back to memory.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If the main path tensor is stored in memory bank before combining with skip connection tensor, then the memory system can process residual block operations, but the memory access latency increases and computational efficiency decreases
Solution Approach 1:
The patent introduces bank-specific buffers as intermediary storage between the memory bank and the combiner circuit. These buffers temporarily hold the main path tensor data, allowing the combiner circuit to retrieve data at optimal speeds without causing memory access conflicts or latency. The buffer acts as a mediator that decouples the memory access timing from the computation timing, thereby improving computational efficiency while maintaining data flow continuity.
Solution Approach 2:
The patent implements preliminary action by pre-loading the main path tensor into bank-specific buffers before the actual combination operation with the skip connection tensor. This pre-positioning of data in the buffers ensures that when the combiner circuit needs to access the main path tensor, the data is already available in fast storage, eliminating the need for real-time memory access during the combination operation and thus reducing memory access latency.
2Productivity
If traditional memory system architecture is used for residual block operations, then implementation is simpler, but element-wise operations between tensors require additional memory access steps
Solution Approach 1:
The patent applies segmentation by dividing the memory system into multiple independent memory banks, each with its own dedicated buffer and combiner circuit. This segmentation allows parallel processing of multiple tensor operations simultaneously, where each bank can independently perform element-wise operations without interfering with other banks. The segmentation of functionality (separate buffers and combiners for each bank) enables higher operation speed while keeping each individual bank's architecture relatively simple and modular.
Solution Approach 2:
The patent implements multi-functionality by designing the memory banks to handle both storage and preliminary data preparation functions. The same memory bank structure is reused across multiple banks with identical buffer and combiner configurations, allowing the system to process different tensor operations in parallel using the same architectural blueprint. This universal design approach enables high operation speed through parallelism while avoiding the complexity of designing entirely separate processing paths for each operation type.
Data Source
AI summary
A memory system comprises a plurality of memory sub-systems, each with a memory bank and other circuit components. For each of the memory sub-systems, a first buffer receives and stores a read-modify-write request (with a read address, a write address and a first operand), a second operand is read from the memory bank at the location specified by the read address, a combiner circuit combines the first operand with the second operand, an activation circuit transforms the output of the combiner circuit, and the output of the activation circuit is stored in the memory bank at the location specified by the write address. The first operand and the write address may be stored in a second buffer while the second operand is read from the memory bank. Further, the output of the activation circuit may be first stored in the first buffer before being stored in the memory bank.