CPU Cache Flushing for GPU Kernel Latency Reduction
Find Innovative SolutionsGenerate Solutions
Solution Overview
Problem
In heterogeneous CPU-GPU systems with unified memory address space, maintaining cache coherency incurs performance latency due to the high number of coherence messages transmitted between processors, leading to inefficiencies and bottlenecks in communication fabric, especially when CPU and GPU cores are on separate dies with longer communication latencies.
Innovation Solution
Implementing techniques such as flushing CPU caches and switching from write-back to write-through cache write policies before launching a GPU kernel to reduce coherence traffic, allowing faster access to main memory and minimizing the need for coherence messaging.
Engineering Contradictions & Design Principles
Engineering Contradiction Analysis
1Reliability
If cache coherency protocols are used to maintain consistent data view between CPU and GPU caches, then data consistency is improved, but system latency increases due to coherence message transmission
Solution Approach 1:
The patent applies preliminary action by flushing CPU caches before GPU kernel execution begins. This proactive clearing of CPU cache entries prevents coherence messages from being generated during GPU memory accesses, thereby maintaining data consistency while avoiding the latency overhead of real-time coherence protocols.
Solution Approach 2:
The patent extracts the coherence management burden from the active execution phase by performing cache flushing as a separate preliminary step. This separates the data consistency guarantee from the GPU execution timeline, eliminating coherence message latency during kernel execution while preserving reliability.
2Reliability
If coherence messages are transmitted between CPU and GPU caches to maintain coherency, then cache consistency is improved, but communication fabric performance deteriorates due to message volume
Solution Approach 1:
By flushing CPU caches before GPU execution, the patent proactively eliminates the source of coherence messages. This preliminary action ensures that GPU memory accesses do not encounter cached data in CPU caches, thereby preventing coherence traffic and maintaining communication fabric throughput.
Solution Approach 2:
The patent converts the potential harm of cache coherence complexity into benefit by using cache flushing as a simplification strategy. Instead of managing complex real-time coherence protocols, the system benefits from a simpler approach that eliminates coherence traffic entirely during GPU execution, improving communication fabric productivity.
3Speed
If CPU caches are used to store data for faster access, then data access speed is improved, but coherence traffic increases when GPUs access the same data
Solution Approach 1:
The patent applies preliminary action by clearing CPU caches before GPU execution. This ensures that GPUs access data directly from main memory without encountering cached copies in CPU caches, thereby eliminating coherence message overhead while maintaining acceptable data access speeds for GPU workloads.
Solution Approach 2:
The patent changes the cache state parameter from 'cached' to 'flushed' before GPU execution. This parameter change transitions the system from a state where CPU caches could cause coherence traffic to a state where direct main memory access is used, eliminating the coherence overhead while adapting to GPU execution requirements.
Data Source
AI summary
A processing system includes at least one central processing unit (CPU) core, at least one graphics processing unit (GPU) core, a main memory, and a coherence directory for maintaining cache coherence. The at least one CPU core receives a CPU cache flush command to flush cache lines stored in cache memory of the at least one CPU core prior to launching a GPU kernel. The coherence directory transfers data associated with a memory access request by the at least one GPU core from the main memory without issuing coherence probes to caches of the at least one CPU core.