Replay Buffer for Memory Fault Handling in Parallel Processing Units
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
Conventional processing units in multithreaded computer systems face significant performance degradation due to lengthy fault handling procedures when memory access faults occur, as they often cancel numerous instructions and wait for page faults to be resolved, leading to unacceptable latency.
Innovation Solution
A computer-implemented method for a multithreaded processing unit that receives and attempts to execute virtual memory transactions, detects page faults, stores transactions in a replay buffer, and causes a stall to inhibit further transactions until the fault is resolved, allowing affected units to replay the faulting transactions once resolved, thereby avoiding cancellation of non-faulting transactions.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If conventional instruction-cancellation fault handling technique is implemented, then memory access faults are resolved, but system performance significantly degrades due to cancellation of thousands of instructions and lengthy fault handling procedures
Solution Approach 1:
The fault handling process is segmented into two independent parts: (1) the fault resolution process that handles the actual memory access fault, and (2) the instruction replay process that re-executes affected instructions. This segmentation allows unaffected instructions to continue executing while only the faulting instructions are stalled and replayed, avoiding the need to cancel thousands of instructions as in conventional approaches.
Solution Approach 2:
The system performs preliminary actions by maintaining a replay buffer that stores instructions for potential replay, and by implementing a fault detection mechanism that identifies only the specific instructions affected by the fault. This preliminary preparation enables the system to quickly resume execution of affected instructions once the fault is resolved, rather than cancelling and re-fetching entire instruction sequences.
2Reliability
If all instructions are cancelled upon memory access fault, then fault resolution is ensured, but latency increases due to waiting for lengthy access fault handling procedures
Solution Approach 1:
The system maintains continuity of useful action by allowing instructions that are not affected by the memory access fault to continue executing without interruption. Only the specific faulting instructions are stalled and placed in the replay buffer, while the vast majority of instructions in the instruction stream continue to execute normally, minimizing the loss of productive work during fault handling.
Solution Approach 2:
The fault handling mechanism applies local quality by selectively identifying and handling only the specific instructions that are affected by the memory access fault, rather than applying a blanket cancellation to all instructions. The replay buffer and fault detection mechanism ensure that only the necessary instructions are stalled and replayed, reducing the overall time loss.
3Reliability
If conventional fault handling is used in highly-parallel multithreaded PPU, then faults are handled, but performance degrades due to frequent cancellation of thousands of instructions across all execution units
Solution Approach 1:
The parallel processing system is segmented into independent execution units, each with its own instruction stream and replay buffer. When a memory access fault occurs in one execution unit, only that specific unit stalls and replays its affected instructions, while all other execution units continue processing their instructions without interruption. This segmentation preserves parallel processing throughput despite fault occurrences.
Solution Approach 2:
The system changes the parameter of fault propagation by implementing per-execution-unit fault handling instead of system-wide fault cancellation. Each execution unit independently tracks its own faulting instructions and maintains its own replay buffer, allowing the system to handle faults locally without affecting the global parallel processing throughput.
Data Source
AI summary
One embodiment of the present invention is a parallel processing unit (PPU) that includes one or more streaming multiprocessors (SMs) and implements a replay unit per SM. Upon detecting a page fault associated with a memory transaction issued by a particular SM, the corresponding replay unit causes the SM, but not any unaffected SMs, to cease issuing new memory transactions. The replay unit then stores the faulting memory transaction and any faulting in-flight memory transaction in a replay buffer. As page faults are resolved, the replay unit replays the memory transactions in the replay buffer—removing successful memory transactions from the replay buffer—until all of the stored memory transactions have successfully executed. Advantageously, the overall performance of the PPU is improved compared to conventional PPUs that, upon detecting a page fault, stop performing memory transactions across all SMs included in the PPU until the fault is resolved.