Cache Bypass Path for Multi-Thread Instruction Fetch
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In multi-threaded data processing systems, particularly in real-time applications, the performance of high priority threads is compromised by the round-robin instruction fetch mechanism, which can lead to significant reduction in overall instruction throughput due to the adverse effects on low priority threads, and existing solutions like replicated or multi-ported caches are costly and energy-inefficient.
Innovation Solution
A cache memory hierarchy with a bypass path that omits a predetermined level for low priority thread access requests, ensuring high priority threads have uninterrupted access and allowing low priority threads to fetch instructions simultaneously without interfering with high priority threads, while also incorporating a shadow fetch buffer to reduce energy consumption.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Ease of operation
If round-robin instruction fetch mechanism is used for multiple threads, then fairness in bandwidth distribution is improved, but performance of high priority threads deteriorates due to lengthened execution time
Solution Approach 1:
The instruction cache is segmented into multiple banks that can be independently accessed. High priority threads can access any bank while low priority threads are restricted to specific banks, enabling simultaneous fetch operations without conflict arbitration that would otherwise delay high priority threads
Solution Approach 2:
Different quality of service is provided to different thread priorities through the bypass path mechanism. Low priority thread accesses to the first cache bank are redirected through a bypass path that omits the first cache level, ensuring high priority threads receive uninterrupted service while low priority threads still obtain instruction fetch capability
2Speed
If full instruction fetch bandwidth is assigned to high priority thread at every cycle, then execution time of high priority thread is minimized, but instruction throughput of low priority threads deteriorates significantly
Solution Approach 1:
The system adds a temporal dimension to bandwidth allocation by enabling simultaneous fetch operations at different cache levels. High priority threads fetch from the first cache bank while low priority threads fetch from the second cache bank through the bypass path, effectively creating parallel instruction fetch streams that increase overall system throughput without compromising high priority thread execution speed
Solution Approach 2:
The bypass path acts as an intermediary mechanism that allows low priority thread instruction fetch requests to coexist with high priority thread requests. By providing an alternative access path that omits the first cache level, the system mediates between competing bandwidth requirements without forcing low priority threads to wait for high priority thread stalls
3Productivity
If instruction cache is made multi-ported to allow independent fetch per thread, then simultaneous fetching capability is improved, but cost and energy consumption deteriorate
Solution Approach 1:
The instruction cache is divided into multiple banks with independent access ports. Each bank can service fetch requests independently, allowing simultaneous instruction fetching for multiple threads without requiring a fully multi-ported cache structure, thereby reducing complexity and energy consumption while maintaining parallel fetch capability
4Productivity
If instruction cache is partitioned into several banks for simultaneous fetch, then parallel fetching capability is improved, but cache access time deteriorates due to bank conflict arbitration
Solution Approach 1:
The system pre-allocates specific cache banks to specific thread priorities to prevent bank conflicts before they occur. Low priority threads are pre-assigned to the second cache bank with bypass path access, while high priority threads have access to the first cache bank. This preliminary assignment eliminates the need for runtime arbitration that would otherwise increase cache access time
Data Source
AI summary
A data processing apparatus is provided wherein processing circuitry executes multiple program threads including at least one high priority thread and at least one lower priority thread. Instructions required by the threads are retrieved from a cache memory hierarchy comprising multiple cache levels. The cache memory hierarchy includes a bypass path for omitting a predetermined level of the cache memory hierarchy when performing a lookup procedure for a required instruction and for bypassing said predetermined level of the cache memory hierarchy when returning said required instruction to said processing circuitry. The bypass path is used by default when the requested instruction is for a lower priority thread.