Interleaving Memory Requests for Neural Network Hardware
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing hardware circuits face inefficiencies in processing multiple small memory access operations across large system memory, leading to increased idle cycles and reduced bandwidth when handling non-linear requests that span various memory banks.
Innovation Solution
The implementation of a hardware circuit with an improved architecture that interleaves memory access requests to optimize data retrieval from multiple memory banks, reducing the need for idle cycles by processing sub-requests across different bank groups, thereby enhancing processing efficiency and bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If multiple small memory access operations are processed sequentially across large system memory, then memory access completeness is ensured, but system bandwidth decreases and idle cycles increase
Solution Approach 1:
The patent segments memory banks into different bank groups and divides memory access requests into sub-requests that can be processed in parallel across different bank groups. This segmentation allows simultaneous processing of multiple sub-requests without conflicts, thereby increasing system bandwidth and reducing idle cycles caused by sequential processing bottlenecks.
Solution Approach 2:
The patent introduces a new dimension of parallelism by processing sub-requests from different bank groups simultaneously rather than sequentially. By utilizing multiple bank groups as separate processing dimensions, the system can execute multiple memory operations in parallel, effectively increasing throughput and reducing the time loss from idle cycles.
2Productivity
If back-to-back read requests are issued to the same memory bank, then data retrieval completeness is achieved, but power consumption increases due to repeated bank activations
Solution Approach 1:
The patent segments consecutive read requests into sub-requests distributed across different bank groups. By doing so, it avoids repeated activations of the same memory bank, thereby reducing power consumption while maintaining data retrieval speed through parallel access to multiple banks.
Solution Approach 2:
The patent merges multiple read requests targeting the same bank into a single coordinated operation across bank groups. This merging strategy reduces redundant bank activations and associated power consumption while maintaining the completeness and speed of data retrieval through efficient request consolidation.
3Productivity
If memory access requests are processed without interleaving, then processing simplicity is maintained, but processing efficiency decreases due to sequential handling
Solution Approach 1:
The patent segments memory access requests into sub-requests that can be independently processed and interleaved across bank groups. This segmentation enables parallel processing improvement while managing complexity through structured subdivision of requests into manageable units that can be handled by existing memory control mechanisms.
Solution Approach 2:
The patent performs preliminary ordering and organization of sub-requests before they are issued to memory banks. This preliminary action ensures that interleaved requests are properly sequenced and managed, improving processing efficiency while controlling complexity through advance preparation rather than complex real-time arbitration.
Data Source
AI summary
Methods, systems, and apparatus, including computer-readable media, are described for interleaving memory requests to accelerate memory accesses at a hardware circuit configured to implement a neural network model. A system generates multiple requests that are processed against a memory of the system. Each request is used to retrieve data from the memory. For each request, the system generates multiple sub-requests based on a respective size of the data to be retrieved using the request. The system generates a sequence of interleaved sub-requests that includes respective sub-requests of a first request interleaved among respective sub-requests of a second request. Based on the sequence of interleaved sub-requests, a module of the system receives respective portions of data accessed from different address locations of the memory. The system processes each of the respective portions of data to generate a neural network inference using the neural network model implemented at the hardware circuit.