Submission Queue Data Prefetching for Lower Memory Access Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional memory sub-systems face challenges in maximizing bandwidth utilization and minimizing latency in highly threaded environments with variable queue depths, particularly in computing systems performing artificial neural network (ANN) and artificial intelligence (AI) tasks, due to unpredictable thread patterns and varying queue depths.
Innovation Solution
Implementing a prefetch mechanism in memory sub-systems with a large fast random access memory capacity, using speculative prefetching to buffer data from slower storage media, and dynamically allocating and deallocating buffers based on access patterns to optimize performance.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If conventional memory sub-systems process commands from submission queues in highly threaded environments, then they can handle multiple concurrent requests, but bandwidth utilization is not maximized and latency increases due to unpredictable thread patterns and variable queue depths
Solution Approach 1:
The memory sub-system performs speculative prefetching by predicting future data access patterns before actual commands are issued. The command execution circuitry anticipates which data will be needed next and pre-loads it into buffers, eliminating waiting time when commands arrive. This preliminary action resolves the contradiction by preparing data in advance, maximizing bandwidth utilization while reducing latency caused by unpredictable thread patterns.
Solution Approach 2:
The system dynamically adjusts prefetching behavior based on observed access patterns and queue depth variations. The command execution circuitry monitors thread behavior and modifies prefetching intensity and target addresses in real-time. This dynamic adaptation allows the system to optimize bandwidth utilization for sequential accesses while avoiding unnecessary prefetching for random patterns, thereby reducing latency without wasting bandwidth.
2Productivity
If the memory sub-system uses a large fast random access memory capacity for prefetching, then bandwidth utilization improves and latency reduces, but device complexity increases
Solution Approach 1:
The command execution circuitry autonomously performs prefetching decisions and buffer management without external control. It self-monitors access patterns, self-determines prefetch targets, and self-manages buffer allocation and deallocation. This self-service capability resolves the contradiction by embedding intelligence directly in the command execution unit, improving performance through aggressive prefetching while avoiding the complexity of external buffer management systems.
Solution Approach 2:
The patent combines the command execution functionality with prefetching logic and buffer management into a single integrated command execution circuitry. Rather than having separate prefetching units and buffer managers, these functions are merged into the existing command execution path. This integration resolves the contradiction by achieving high performance through unified functionality while reducing overall device complexity through consolidation.
3Productivity
If speculative prefetching is implemented to predict data access patterns, then bandwidth utilization increases and latency decreases, but the difficulty of detecting and measuring access patterns increases
Solution Approach 1:
The command execution circuitry performs prefetching for a limited number of future addresses beyond what is strictly necessary. Instead of attempting to perfectly predict the exact next access point, it prefetches a range of potential targets. This partial/excessive approach resolves the contradiction by achieving good bandwidth utilization through conservative over-prefetching while avoiding the complexity of precise pattern detection, as the system accepts some redundancy in exchange for simpler detection logic.
Data Source
AI summary
A memory sub-system, including: a random access memory; a storage medium having a storage capacity accessible to a host system through commands communicated via a plurality of submission queues to the memory sub-system; and a controller. The controller is configured to: retrieve, from a first submission queue among the plurality of submission queues, a first command configured with a first address to access the storage medium; determine, based at least in part on the first command, that a second address configured in a second command following the first command in communication via the first submission queue from the host system to the memory sub-system is predictable; predict a third address according to the first address configured in the first command; and retrieve, from the storage medium and according to the third address, a data chunk into the random access memory, before the second command is retrieved.