Dual Query Stage Pipeline Hides Cache Miss Latency
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Data processing systems face inefficiencies due to cache misses, which lead to delayed instruction processing and excessive buffering requirements, as they need to query the cache well in advance for data, resulting in high latency and increased circuit size, power consumption, and cost.
Innovation Solution
Implementing a dual query stage system with a main query stage and a buffer query stage, where program instructions with cache misses are stored in a buffer, allowing concurrent query requests to the cache memory, reducing the need for large buffers and hiding memory latency.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Loss of time
If data is queried from cache memory significantly in advance of when it is required, then cache miss latency is hidden, but buffering requirements become excessively large
Solution Approach 1:
The query stage is divided into two separate stages: a main query stage that operates in parallel with instruction fetch, and a buffer query stage that operates on instructions in the buffer. This segmentation allows the system to hide cache miss latency without requiring excessively large buffers, as each stage handles a specific portion of the query workload independently.
Solution Approach 2:
The main query stage performs preliminary cache queries in parallel with instruction fetch operations, preparing data in advance before it is actually needed by subsequent pipeline stages. This preliminary action hides cache miss latency without requiring the entire instruction pipeline to stall, thereby reducing the buffering requirements compared to traditional advance querying approaches.
2Loss of time
If a single query stage is used, then device complexity is low, but cache miss latency cannot be effectively hidden
Solution Approach 1:
The query functionality is segmented into two distinct query stages with different responsibilities: the main query stage handles queries for instructions currently in the pipeline, while the buffer query stage handles queries for instructions in the buffer. This segmentation enables effective latency hiding while maintaining manageable complexity through clear separation of concerns.
Solution Approach 2:
The main query stage and buffer query stage are merged into a unified parallel processing structure where both stages operate simultaneously and independently. This merging allows the system to hide cache miss latency effectively while keeping each individual query stage relatively simple, balancing complexity and performance.
3Productivity
If program instructions are buffered until cache data is available, then instruction processing is maintained, but buffer size increases circuit size and power consumption
Solution Approach 1:
The instruction processing stream is segmented into two paths: instructions that hit in the cache proceed directly to execution, while instructions that miss are redirected to the buffer query stage. This segmentation ensures that only the necessary instructions are buffered, minimizing buffer size and reducing power consumption while maintaining instruction processing throughput.
Solution Approach 2:
The buffer query stage continuously operates in parallel with the main query stage, ensuring that cache queries are performed without interruption. This continuous operation maintains instruction processing throughput by preventing pipeline stalls, while the selective buffering approach minimizes the buffer size and associated power consumption.
Data Source
AI summary
A processing pipeline 6, 8, 10, 12 is provided with a main query stage 20 and a fetch stage 22. A buffer 24 stores program instructions which have missed within a cache memory 14. Query generation circuitry within the main query stage 20 and within a buffer query stage 26 serve to concurrently generate a main query request and a buffer query request sent to the cache memory 14. The cache memory returns a main query response and a buffer query response. Arbitration circuitry 28 controls multiplexers 30, 32 and 34 to direct the program instruction at the main query stage 20, and the program instruction stored within the buffer 24 and the buffer query stage 26 to pass either to the fetch stage 22 or to the buffer 24. The multiplexer 30 can also select a new instruction to be passed to the main query stage 20.