Staged Memory Scheduler Batching Requests for CPU Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In modern chip multiprocessor systems, memory interference between CPU and GPU cores leads to significant performance degradation due to the large volume of memory requests from GPUs, which occupy a significant fraction of the request buffer, limiting the visibility and management of CPU memory behaviors and causing slowdowns for non-memory-intensive applications.
Innovation Solution
A staged memory scheduler with a batch unit and batch scheduler that decouples memory scheduling into decentralized source queues, forming batches of memory requests and prioritizing them based on latency-critical applications while ensuring reasonable progress for bandwidth-intensive applications, using probabilistic policies and bypass logic for low-intensity sources.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Device complexity
If a centralized request buffer is used to manage memory requests from multiple sources, then memory scheduling can be simplified, but GPU requests occupy most buffer entries causing CPU requests to be starved and increasing latency for CPU applications
Solution Approach 1:
The patent divides the centralized request buffer into multiple separate source queues, one for each memory request source (CPU0, CPU1, GPU, etc.). This segmentation prevents any single source from dominating the buffer and ensures that CPU requests from different cores are visible and schedulable independently of GPU request volume.
Solution Approach 2:
The patent introduces source queues as intermediary structures between the memory request sources and the memory scheduler. These source queues act as buffers that decouple the high-volume GPU requests from the CPU requests, allowing the scheduler to manage CPU requests without being overwhelmed by GPU traffic.
2Speed
If memory requests are serviced individually without batching, then request response time is reduced, but row-buffer locality is not preserved leading to increased memory access latency
Solution Approach 1:
The patent performs preliminary actions by gathering multiple memory requests into batches before servicing them. The batch unit collects requests that target the same memory row and processes them together, ensuring that subsequent requests in the batch benefit from row-buffer hits rather than requiring costly row activations.
Solution Approach 2:
The patent merges multiple individual memory requests into consolidated batches based on their target row. By combining requests that access the same memory row, the system preserves row-buffer locality and reduces the total number of row activations required, thereby decreasing overall memory access latency.
3Loss of time
If the request buffer size is increased to accommodate more CPU requests, then CPU request visibility improves, but the buffer occupies more die area and increases controller complexity
Solution Approach 1:
Instead of using one large centralized buffer, the patent segments the buffer into multiple smaller source queues. Each source queue maintains a manageable size while collectively providing sufficient capacity for CPU requests from multiple cores. This segmentation reduces the total die area required compared to a single large buffer.
Solution Approach 2:
The patent assigns different buffer characteristics to different source queues based on their specific needs. Each source queue can be optimized independently for its particular traffic pattern, allowing for more efficient use of die area while ensuring adequate visibility for CPU requests from each core.
Data Source
AI summary
A memory controller includes a batch unit, a batch scheduler, and a memory command scheduler. The batch unit includes a plurality of source queues for receiving memory requests from a plurality of sources. Each source is associated with a selected one of the source queues. The batch unit is operable to generate batches of memory requests in the source queues. The batch scheduler is operable to select a batch from one of the source queues. The memory command scheduler is operable to receive the selected batch from the batch scheduler and issue the memory requests in the selected batch to a memory interfacing with the memory controller.