Memory Bank Read-Modify-Write Buffering for CNN Residual Blocks
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing memory systems for neural networks, particularly Convolutional Neural Networks (CNNs), face inefficiencies in processing residual block operations due to the need for preliminary storage of tensors before combination, which hinders concurrent processing and increases memory access latency.
Innovation Solution
A memory system architecture that processes read-modify-write requests by routing these requests to appropriate memory banks with bank-specific buffers and combiner circuits for element-wise operations, allowing temporary storage of operands and applying activation functions without prior storage of the main path tensor, enabling concurrent processing and reduced latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If the main path tensor is first stored in memory bank prior to combination with skip connection tensor, then the data can be retrieved and combined, but this preliminary storage increases memory access latency and hinders concurrent processing
Solution Approach 1:
The patent applies preliminary action by performing the read operation for the skip connection tensor before the write operation for the main path tensor. The system reads the skip connection tensor from memory first, then performs the element-wise addition with the main path tensor, and finally writes the result back to memory. This reordering eliminates the need to store the main path tensor in memory before combination, reducing memory access latency while ensuring data retrieval accuracy.
Solution Approach 2:
The patent introduces an intermediary buffer structure that temporarily holds the skip connection tensor during the combination operation. This buffer acts as a mediator between the read operation and the combination operation, allowing the main path tensor to be processed without preliminary storage in the main memory bank, thereby reducing memory access latency while maintaining data integrity.
2Reliability
If preliminary storage of tensors is performed before combination, then data integrity is ensured, but concurrent processing capability is reduced
Solution Approach 1:
The patent reorders operations to perform the read of the skip connection tensor before the write of the main path tensor result. This preliminary read action allows the combination operation to proceed without waiting for preliminary storage of the main path tensor, enabling concurrent processing of multiple residual blocks while maintaining data integrity through the element-wise addition operation.
Solution Approach 2:
The patent segments the residual block processing into independent parallel operations across multiple memory banks. Each memory bank can process different residual blocks concurrently, with each bank performing the read-modify-write sequence independently. This segmentation enables concurrent processing while ensuring data integrity within each segmented operation.
3Manufacturing precision
If traditional memory access sequence is used (store main path tensor, then retrieve and combine), then processing correctness is maintained, but memory access efficiency decreases
Solution Approach 1:
The patent performs the read operation for the skip connection tensor as a preliminary action before the write operation for the main path tensor result. This reordering maintains processing correctness by ensuring both tensors are available for the element-wise addition while improving memory access efficiency by eliminating the preliminary storage step for the main path tensor.
Data Source
AI summary
A memory system comprises a plurality of memory sub-systems, each with a memory bank and other circuit components. For each of the memory sub-systems, a first buffer receives and stores a read-modify-write request (with a read address, a write address and a first operand), a second operand is read from the memory bank at the location specified by the read address, a combiner circuit combines the first operand with the second operand, an activation circuit transforms the output of the combiner circuit, and the output of the activation circuit is stored in the memory bank at the location specified by the write address. The first operand and the write address may be stored in a second buffer while the second operand is read from the memory bank. Further, the output of the activation circuit may be first stored in the first buffer before being stored in the memory bank.