On-Chip Cache Request Reordering to Let Hits Overtake Misses
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing memory hierarchies suffer from increased latency due to cache misses, which can stall subsequent memory requests, leading to inefficiencies and higher power consumption.
Innovation Solution
An on-chip cache system that receives memory requests and reorders them based on age and availability of data, allowing cache hits to overtake misses, eliminating the need for intermediate buffers and reducing latency by managing pending requests through a data structure like a circular buffer.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If memory requests are processed in strict order, then correctness is maintained, but latency increases due to cache misses stalling subsequent requests
Solution Approach 1:
The patent segments memory requests into different categories (cache hits and cache misses) and processes them through separate pathways. The reorder buffer segments the processing stream, allowing cache hits to be executed immediately while cache misses are handled separately, thus reducing latency without compromising correctness.
Solution Approach 2:
The patent performs preliminary actions by pre-executing cache hit requests before cache miss requests are fully processed. The reorder buffer holds and reorders requests, allowing faster cache hit operations to proceed ahead of slower cache miss operations, effectively performing useful work in advance.
2Loss of time
If intermediate buffers are added to reorder memory requests, then latency is reduced, but device complexity increases
Solution Approach 1:
The patent merges the reorder buffer functionality with the existing cache structure, combining multiple functions (request buffering, reordering, and execution control) into a single integrated component. This reduces the need for separate intermediate buffers and simplifies the overall system architecture.
Solution Approach 2:
The reorder buffer serves multiple functions: it buffers pending requests, reorders them based on completion status, and controls execution timing. This multi-functionality eliminates the need for separate dedicated buffers for each function, reducing overall device complexity while maintaining latency benefits.
Data Source
AI summary
An on-chip cache is described which receives memory requests and in the event of a cache miss, the cache generates memory requests to a lower level in the memory hierarchy (e.g. to a lower level cache or an external memory). Data returned to the on-chip cache in response to the generated memory requests may be received out-of-order. An instruction scheduler in the on-chip cache stores pending received memory requests and effects the re-ordering by selecting a sequence of pending memory requests for execution such that pending requests relating to an identical cache line are executed in age order and pending requests relating to different cache lines are executed in an order dependent upon when data relating to the different cache lines is returned. The memory requests which are received may be received from another, lower level on-chip cache or from registers.