Multicore Instruction Prefetch Pipeline for Coherence-Free SIMD Delivery
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Existing SIMD processing units face challenges in efficiently providing instructions due to the high cost of adding instruction memory to each tile and the complexity of full cache coherent solutions, which are impractical for cross-lane processing units (XPUs).
Innovation Solution
An instruction prefetch pipeline architecture is implemented, comprising a shared instruction memory (TiMem) and independent instruction buffers (iBuf) per core, with a prefetch unit and router, optimizing instruction delivery through prefetching and filtering to reduce redundant requests and enhance bandwidth.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Productivity
If a full cache coherent solution is deployed in CPUs, then instruction delivery efficiency is improved, but device complexity increases significantly
Solution Approach 1:
The system segments the instruction memory into multiple task instruction memories (one per data lane) and separates the prefetch function into independent prefetch units for each tile. This segmentation allows each component to operate independently without requiring complex cache coherence protocols across the entire system, thus improving instruction delivery efficiency while reducing device complexity.
Solution Approach 2:
The prefetch unit performs preliminary action by fetching instructions from task instruction memories ahead of time and storing them in local instruction buffers before the compute cores need them. This advance preparation eliminates the need for complex cache coherence mechanisms during actual instruction execution, as instructions are already available in the local buffers when needed.
2Speed
If each tile has its own instruction memory, then instruction access speed is improved, but manufacturing cost increases
Solution Approach 1:
The system merges multiple task instruction memories into a single shared instruction memory resource that serves all data lanes. The instruction router intelligently directs prefetch requests to the appropriate task instruction memory based on the data lane identifier, allowing multiple tiles to share the same physical memory resources while maintaining fast access speeds through parallel prefetch operations.
Solution Approach 2:
The shared instruction memory is designed to serve multiple functions: it can be accessed by any data lane through the instruction router, and each task instruction memory within it can service multiple tiles. This multi-functionality reduces the total number of memory components needed, lowering manufacturing costs while maintaining fast instruction access through efficient routing and parallel access paths.
3Loss of time
If instructions are prefetched ahead of time, then cold cache misses are reduced, but bandwidth requirements increase
Solution Approach 1:
The prefetch bandwidth is segmented into multiple independent channels, one for each data lane. Each prefetch unit operates independently on its assigned data lane, fetching instructions in parallel. This segmentation distributes the total bandwidth requirement across multiple channels, reducing the peak bandwidth demand on any single channel while effectively prefetching all needed instructions and eliminating cold cache misses.
Solution Approach 2:
The prefetch unit implements partial action by fetching only the specific instructions needed for the current and near-future operations of each data lane, rather than prefetching all possible instructions. The prefetch window is limited to a reasonable ahead-of-time horizon, fetching just enough instructions to keep the instruction buffer filled without unnecessarily consuming excessive bandwidth. This selective prefetching reduces bandwidth consumption while still eliminating cold cache misses for the actually executed instructions.
Data Source
AI summary
Aspects of the disclosure are directed to methods, systems, and apparatuses using an instruction prefetch pipeline architecture that provides good performance without the complexity of a full cache coherent solution deployed in conventional CPUs. The architecture can include components which can be used to construct an instruction prefetch pipeline, including instruction memory (TiMem), instruction buffer (iBuf), a prefetch unit, and an instruction router.